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Executive  Summary 


Capability  Maturity  Model  Integration  (CMMI)  was  developed  as  an  integrated  approach  to 
internal  process  improvement  in  a  broad  range  of  organizations.  CMMI  for  Development  (CMMI- 
DEV),  Version  1.2,  consists  of  a  collection  of  best  practices  that  address  product  development  and 
maintenance  activities  that  cover  the  product  lifecycle,  from  conception  through  delivery  and 
sustainment. 

This  guidebook  is  designed  to  help  an  acquirer  benefit  from  a  supplier’s  use  of  CMMI-DEV  while 
avoiding  the  pitfalls  associated  with  unrealistic  expectations.  It  provides  insight  on  the  following: 

•  Interpreting  a  supplier’s  claims  of  achieving  a  CMMI  rating 

•  The  significance  of  the  improvements  to  the  CMMI  Product  Suite  in  v  1.2 

•  How  to  request,  understand,  interpret,  and  use  the  results  of  a  supplier’s  CMMI  appraisals 

•  How  to  determine  which  process  areas  (PAs)  are  critical  to  the  success  of  the  program1 

•  Methods  that  leverage  a  supplier’s  process  improvement  initiatives 

•  How  to  gather  and  interpret  information  for  effectively  monitoring  supplier  processes 
throughout  the  execution  of  an  acquisition  program 

This  guidebook  also  describes  CMMI  fundamentals  that  acquirers  must  understand  to  effectively 
use  information  obtained  from  a  supplier’s  CMMI-DEV  efforts  on  development  programs.  It 
includes  explanations  of  capability  and  maturity  levels  and  the  differences  between  the  two 
CMMI  representations — continuous  and  staged.  It  also  explains  some  of  the  more  obscure 
elements  of  CMMI,  such  as  equivalent  staging  and  high  maturity  and  capability  levels  (levels  4 
and  5).  The  terms  and  concepts  explained  are  ones  acquirers  may  encounter  in  proposals  and  in 
everyday  dealings  with  suppliers.  This  guidebook  intends  to  demystify  these  terms  and  concepts. 

Acquirers  and  users  of  CMMI-DEV,  in  general,  should  be  cautioned  that  high  capability 
and  maturity  level  ratings  alone  do  not  guarantee  program  success.  The  U.S.  Department  of 
Defense  (DoD)  has  never  promulgated  a  policy  requiring  adherence  to  any  CMMI  maturity  level 
rating  but  rather  promotes  CMMI  as  a  tool  for  internal  process  improvement.  This  guidebook 
helps  clarify  what  high  capability  and  maturity  level  ratings  can  and  cannot  do  for  a  development 
program. 

For  more  information  on  the  CMMI  Product  Suite  refer  to  the  CMMI  Web  site  at 
http://www.sei.cmu.edu/cmmi/. 


1  This  report  uses  the  term  program  in  place  of  the  CMMI  model  standard  term  project. 
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Abstract 


This  guidebook  is  designed  to  help  acquisition  organizations  benefit  from  their  suppliers’  use  of 
CMMI  for  Development,  which  is  a  collection  of  best  practices  that  addresses  product 
development  and  maintenance  activities  throughout  the  product  lifecycle.  This  guidebook  also 
helps  acquisition  organizations  avoid  problems  that  can  result  from  unrealistic  expectations. 

High  capability  and  maturity  level  ratings  alone  do  not  guarantee  program  success.  This 
guidebook  helps  clarify  what  those  ratings  can  and  cannot  do  for  a  development  program.  It 
describes  how  acquirers  can 

•  interpret  suppliers’  claims  of  achieving  a  CMMI  rating 

•  request,  understand,  interpret,  and  use  supplier  appraisal  results,  and 

•  apply  methods  that  leverage  a  supplier’s  process  improvement  initiatives. 
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1  Introduction 


2 

This  guidebook  is  designed  to  help  acquisition  organizations  pose  the  appropriate  CMMI-related 
questions  to  their  suppliers  and  interpret  supplier  responses  to  help  identify  and  evaluate  the  risks 
that  may  be  associated  with  a  particular  supplier.  It  also  is  designed  to  provide  the  acquisition 
organization  with  approaches  for  gathering  and  interpreting  supplier  data  throughout  the 
acquisition  process  to  effectively  monitor  supplier  processes  that  are  part  of  the  acquisition 
program  after  contract  award.  This  guidebook  deals  exclusively  with  supplier  processes.  It  does 
not  address  issues  of  process  application,  process  capability,  or  organizational  maturity  within  the 
acquisition  organization. 


Acquisition 

Organization 


Processes 


Figure  1:  Guidebook  Focus  —  Understanding  Supplier  Processes 

CMMI-DEV  contains  process  requirements  based  on  industry  best  practices  that  can  be  used  to 
guide  organizations  in  pursuit  of  improving  their  development  processes.  Organizations  that 
concentrate  on  process  improvement  demonstrate  reductions  in  delivered  defects,  improved  speed 
of  development,  and  fewer  post-delivery  failures.  Given  those  results,  organizations  that 
implement  CMMI-based  process  improvement  may  provide  a  lower  risk  in  development  over 
those  that  do  not. 

CMMI-DEV  provides  a  reference  against  which  organizations  can  be  appraised  using  formal 
appraisal  techniques.  Using  CMMI  appraisal  data  for  evaluation  in  acquisitions  can  be  an 
effective  means  of  identifying  and  evaluating  risk  within  the  development  portion  of  an 
acquisition.  Acquisition  organizations  are  especially  adept  at  identifying  and  evaluating  risks,  and 


2  An  acquisition  organization  may  be  a  government  acquirer  (e.g.,  Program  Management  Office  [PMO],  Joint  Pro¬ 
gram  Office  [JPO]),  a  government  contractor  serving  as  an  acquirer  for  the  government,  or  the  acquisition  organi¬ 
zation  within  a  commercial  enterprise.  This  guidebook  is  written  for  all  of  these  audiences,  and  the  term  acquirer 
represents  all  of  them. 

3  Since  this  guidebook  is  part  of  the  vl  .2  CMMI  Product  Suite,  it  refers  to  CMMI-DEV  [SEI  2006a],  CMMI  Version 
1.1  model  titles  included  Systems  Engineering  (SE),  Software  Engineering  (SW)  or  both  (SE/SW)  along  with  addi¬ 
tions  for  Integrated  Product  and  Process  Development  (IPPD)  and  Supplier  Sourcing  (SS)  [SEI  2002a,  SEI 
2002b],  Version  1 .2  has  an  IPPD  addition  but  SS  has  been  incorporated  into  the  basic  model. 
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have  always  considered  development  as  a  risk  area.  CMMI  appraisal  data,  obtained  using  the 
Standard  CMMI  Appraisal  Method  for  Process  Improvement  (SCAMPIsm),  identify  and  evaluate 
development  processes  that  organizations  defined  and  utilized  in  the  execution  of  their  programs. 
Thus,  this  data  can  help  to  identify  the  degree  of  risk  related  to  a  supplier’s  organizational 
processes  used  for  product  development. 

CMMI-DEV  and  its  appraisal  method  provide  an  approach  for  determining  how  well  the 
appraised  organization  has  employed  its  processes  on  the  programs  evaluated  as  part  of  the 
appraisal.  This  approach  can  indicate  the  degree  of  risk  (or  lack  of  risk)  that  these  same  processes 
might  contribute  when  implemented  on  a  new  program.  This  indication  is  only  valid,  however,  if 
the  supplier  actually  uses  the  appraised  processes  on  the  program  at  hand.  Implementation  of 
these  processes  should  begin  before  contract  award  (in  the  capture  phase),  and  continue  in  earnest 
post  award  in  order  to  reap  the  benefits.  This  guidebook  provides  guidance  to  help  ensure  that  this 
consistent  use  of  appraised  processes  by  the  supplier  happens. 

Sections  1.1  and  1.2  provide  foundational  material  on  CMMI-DEV  and  CMMI  appraisals. 

Readers  familiar  with  CMMI-DEV  and  the  appraisal  method  may  skip  these  sections.  Section  1.3 
includes  information  to  help  acquirers  interpret  the  results  of  CMMI  appraisals. 

1.1  CMMI  FUNDAMENTALS 

CMMI-DEV  consists  of  a  set  of  process  requirements  based  on  industry  best  practices  that  are 
organized  into  22  different  process  areas  across  four  categories,  as  shown  in  Table  1.  (The  process 
areas  related  to  Project  Management,  Engineering,  and  Support  are  discussed  in  Chapter  2.)  The 
Process  Management  process  areas  are  used  at  the  organization  level  to  define,  deploy,  and 
improve  processes  across  the  organization  and  are  important  because  they  play  a  large  part  in  how 
effective  the  organization  deploys  its  processes  on  a  new  program. 


Table  1 :  CMMI-DEV  Process  Areas 


Process  Category 

Project  Management 

Engineering 

Support 

Process  Management 

Project  Planning  (PP) 

Requirements 

Management  (REQM) 

Configuration 

Management  (CM) 

Organizational  Process 
Focus  (OPF) 

Project  Monitoring  and 
Control  (PMC) 

Requirements 

Development  (RD) 

Process  and  Product 

Quality  Assurance  (PPQA) 

Organizational  Process 
Definition  (OPD) 

Supplier  Agreement 
Management  (SAM) 

Technical  Solution  (TS) 

Measurement  and 

Analysis  (MA) 

Organizational  Training 
(OT) 

Integrated  Project 
Management  (IPM) 

Product  Integration  (PI) 

Decision  Analysis  and 
Resolution  (DAR) 

Organizational  Process 
Performance  (OPP) 

Risk  Management  (RSKM) 

Verification  (VER) 

Causal  Analysis  and 
Resolution  (CAR) 

Organizational  Innovation 
and  Deployment  (OID) 

Quantitative  Project 
Management  (QPM) 

Validation  (VAL) 

CMMI-DEV  does  not  dictate  specific  processes,  but  rather  defines  the  best  practices  that  suppliers 
incorporate  into  their  development  processes.  The  degree  to  which  an  organization’s  development 
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processes  conform  to  CMMI-DEV  is  measured  using  an  appraisal  on  a  representative  sample  of 
programs  in  the  appraised  organization.  Many  organizations  use  appraisals  as  a  means  of 
assessing  their  process  capabilities  and  guiding  process  improvement  activities. 

An  appraisal  can  result  in  a  capability  level  profile  across  a  number  of  process  areas  or  a  maturity 
level  rating  for  the  organization,  depending  on  the  model  representation  used.  CMMI-DEV  has 
two  representations  that  are  used  for  appraisals — continuous  and  staged — that  lead  to  capability 
level  and  maturity  level  ratings,  respectively.  The  staged  representation  predefines  the  appraisal 
structure  for  each  grouping  of  process  areas,  while  an  appraisal  using  the  continuous 
representation  appraises  each  selected  process  area  independently.  Organizations  may  choose  one 
representation  over  the  other  for  a  variety  of  reasons,  including  the  current  state  of  ongoing 
improvement  initiatives,  the  supplier’s  historical  familiarity  with  a  particular  approach,  and 
perceived  business  needs  and  objectives. 

In  general,  an  organization’s  progress  in  defining  and  improving  its  processes  is  measured  by 
numbers  1—5  representing  levels  of  capability  or  maturity.  Level  1  is  the  lowest,  meaning  that  the 
process  is  being  performed  but  in  an  ad  hoc  manner.  Regardless  of  the  representation  used,  the 
capability  and  maturity  levels  of  interest  are  2-5,  indicating  to  increasing  degrees  the 
sophistication  and  institutionalization  of  the  process  improvement  efforts  in  the  organization: 

•  Level  2  indicates  managed  processes 

•  Level  3  indicates  defined  program  processes  tailored  from  the  organization’s  set  of  standard 
processes 

•  Level  4  indicates  quantitatively  managed  processes  and  subprocesses 

•  Level  5  indicates  optimizing  processes  and  subprocesses 

The  continuous  representation  of  CMMI-DEV  measures  process  capability  within  each  process 
area.  It  allows  an  organization  to  choose  which  process  areas  to  appraise  based  on  its  business 
objectives  and  process  improvement  goals.  An  appraisal  using  the  continuous  representation 
consists  of  a  capability  level  profile  showing  the  capability  level  achieved  within  each  chosen 
process  area. 

The  staged  representation  specifies  sets  of  process  areas  in  an  ordered  grouping  of  five  maturity 
levels  and  predefines  which  process  areas  must  be  successfully  appraised  to  achieve  a  maturity 
level  rating.  An  appraisal  for  the  staged  representation  results  in  a  maturity  level  rating, 
interpreted  as  follows: 

•  Maturity  level  2  indicates  that  the  organization  has  achieved  capability  level  2  for  each 
process  area  at  maturity  level  2.  Maturity  level  2  is  primarily  demonstrated  at  the  program 
level  and  focuses  on  Project  Management  and  Support  process  areas. 

•  Maturity  level  3  indicates  that  the  organization  has  achieved  capability  level  3  for  all 
maturity  level  2  and  maturity  level  3  process  areas,  which  include  Engineering  and  Process 
Management  process  areas.  It  also  indicates  that  the  organization  has  adopted  a  process 
focus  at  an  organizational  level  and  has  a  set  of  standard  processes  that  can  be  tailored  for 
specific  programs. 
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•  Maturity  level  4  indicates  that  the  organization  has  demonstrated  that  it  quantitatively 
manages  selected  processes  (program  and  organizational  activities  that  are  deemed  important 
and  are  consistent  with  business  objectives)  and  that  it  has  achieved  capability  level  3  on  all 
maturity  level  2,  3,  and  4  process  areas. 

•  Maturity  level  5  indicates  that  the  organization  has  demonstrated  that  it  has  used  its 
measurement  data  to  improve  and  optimize  selected  processes  and  subprocesses,  and  that  it 
has  achieved  capability  level  3  on  all  process  areas  in  the  model. 

Organizations  that  use  the  continuous  representation  can  convert  their  appraised  process  area 
results  into  an  organizational  maturity  level  rating  using  equivalent  staging.  Figure  2,  adapted 
from  CMMI-DEV,  illustrates  the  grouping  of  process  areas  into  the  maturity  levels  of  the  staged 
representation,  the  rules  for  equivalent  staging,  and  what  constitutes  high  maturity. 

Although  high  maturity  is  not  defined  in  CMMI-DEV,  when  practitioners  of  CMMI  refer  to  high 
maturity,  they  are  referring  to  behaviors  associated  with  organizations  performing  at  maturity 
level  4  or  5,  or  having  process  areas  performing  at  capability  levels  4  or  5.  Basically,  such 
behavior  entails  quantitatively  managing  and  optimizing  a  limited  number  of  processes  or 
subprocesses. 

The  grouping  of  process  areas  into  maturity  levels  is  no  indication  of  their  relative  importance.  It 
is  merely  illustrating  a  path  or  sequence  for  process  improvement. 
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Name 

Abbr 

ML 

CL1  CL2 

CL3 

CL4  CL5 

Requirements  Management 

REQM 

2 

High 

Project  Planning 

PP 

2 

Maturity 

Project  Monitoring  and  Control 

PMC 

2 

Supplier  Agreement 
Management 

SAM 

2 

Measurement  and  Analysis 

MA 

2 

Process  and  Product  Quality 
Assurance 

PPQA 

2 

Configuration  Management 

CM 

2 

Requirements  Development 

RD 

3 

Technical  Solution 

TS 

3 

Product  Integration 

PI 

3 

Verification 

VER 

3 

Validation 

VAL 

3 

Organizational  Process  Focus 

OPF 

3 

Organizational  Process 

Definition  +IPPD 

OPD 

+IPPD 

3 

Organizational  Training 

OT 

3 

Integrated  Project 

Management  +IPPD 

IPM 

+IPPD 

3 

Risk  Management 

RSKM 

3 

Decision  Analysis  and 

Resolution 

DAR 

3 

Organizational  Process 
Performance 

OPP 

4 

High  Maturity 

Quantitative  Project 
Management 

QPM 

4 

Organizational  Innovation  and 
Deployment 

OID 

5 

High  Maturity 

Causal  Analysis  and 

Resolution 

CAR 

5 

Figure  2:  Illustration  of  Maturity  Level  Groupings,  Equivalent  Staging,  and  High  Maturity 

1.2  CMMI  APPRAISALS 

Three  types  (generally  referred  to  as  Classes)  of  the  SCAMPI  appraisal  method  are  defined:  Class 
A,  Class  B,  and  Class  C.  Each  Class  has  a  different  purpose,  but  all  are  capable  of  identifying 
process-related  strengths,  weaknesses,  and  risks.  The  appraisal  team  examines  one  or  more 
programs  within  the  organization  for  compliance  to  the  portions  of  the  CMMI  model  specified  in 
the  appraisal  scope.  The  appraisal  process  generally  involves  the  review  of  process  documentation 
(e.g.,  policies  and  procedures),  examination  of  process  artifacts  (e.g.,  work  products),  and 
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interviews  with  staff  to  ascertain  the  degree  of  process  deployment  within  the  organization. 
Irrespective  of  the  representation  of  the  model  the  organization  selected,  the  SCAMPI  appraisal 
method  applies. 

•  SCAMPI  Class  A  is  the  only  formal  appraisal  type  available  for  official  capability  or 
maturity  level  ratings.  Suppliers  that  claim  a  maturity  or  capability  level  rating  can  do  so 
only  as  a  result  of  a  SCAMPI  Class  A  appraisal. 

•  SCAMPI  Class  B  generally  is  used  for  internal  process  improvement,  as  it  does  not  result  in 
a  maturity  level  or  capability  level  rating.  The  SCAMPI  Class  B  appraisal  identifies  gaps  in 
process  implementation  and  is  commonly  referred  to  as  a  gap  analysis. 

•  SCAMPI  Class  C  (commonly  called  a  “quick  look”)  is  less  comprehensive,  but  more 
flexible  than  the  SCAMPI  Class  B,  and  can  be  performed  with  a  small  team.  There  are  fewer 
evidentiary  requirements  for  a  SCAMPI  Class  C  than  for  a  SCAMPI  Class  B  appraisal. 
Often,  a  SCAMPI-C  appraisal  is  a  document  or  artifact  review  without  staff  member 
interviews.  It  primarily  identifies  gaps  in  process  implementation. 

Conducting  a  SCAMPI  Class  A  appraisal  requires  the  participation  of  a  SCAMPI  Lead 
AppraiserSM  authorized  by  the  Software  Engineering  Institute  (SEI).  With  the  latest  release  of 
CMMI-DEV  (vl.2),  the  Lead  Appraiser  for  a  SCAMPI  Class  A  must  be  independent  (i.e.,  from 
an  external,  third  party  organization).  Qualified  Lead  Appraisers  are  listed  in  the  SEI  Partner 
Network  Directory  [SEI  1],  Conducting  a  SCAMPI  Class  B  or  SCAMPI  Class  C  appraisal 
requires  a  trained  and  qualified  appraisal  Team  Leader  authorized  by  the  SEI.  The  Team  Leader 
for  these  appraisals  is  not  required  to  be  from  an  external  organization.  Qualified  SCAMPI  Class 
B  and  C  Team  Leaders  also  are  listed  in  the  SEI  Partner  Network  Directory.  Both  SCAMPI  Lead 
Appraisers  and  Team  Leaders  will  be  referred  to  throughout  this  document  as  appraisal  leads. 

Appraisal  team  members  that  completed  the  training  requirements  delineated  by  the  SEI  may  be 
drawn  from  within  the  appraised  organization  or  external  organizations.  Generally,  an 
organization  conducting  a  SCAMPI  Class  A  appraisal  may  include  more  external  than  internal 
staff  members  to  clearly  demonstrate  appraisal  independence  to  bolster  any  maturity  or  capability 
level  claim. 

The  results  of  SCAMPI  Class  A  appraisals  are  documented  in  an  Appraisal  Disclosure  Statement 
(ADS)  and  an  appraisal  findings  document.  The  Lead  Appraiser  provides  these  documents  to  the 
organization  (i.e.,  the  appraisal  sponsor)  and  to  the  SEI  for  validation.  Until  the  SEI  validates 
appraisal  results,  any  claims  made  by  suppliers  about  levels  are  not  valid. 

For  a  SCAMPI  A  appraisal,  many  organizations  also  permit  the  appraisal  results  to  be  publicly 
posted  on  the  SEI  Published  Appraisal  Report  Site  (PARS)  [SEI  2].  The  acquirer  should  always 
check  the  PARS  to  validate  a  claim,  since  only  validated  results  are  posted  on  PARS.  With  the 
publication  of  version  1.2,  all  U.S.  Department  of  Defense  (DoD)  contractor  ADSs  will  be  posted 
on  the  PARS  site.  Version  1.2  of  the  ADS  significantly  improved  the  reporting  requirements  over 
the  previous  version  making  the  ADS  particularly  useful  to  acquirers  in  determining  the  breadth 
of  an  appraisal.  While  appraisals  conducted  prior  to  vl.2  do  not  have  the  requirement  to  post  the 
ADS  to  PARS,  most  organizations  will  make  the  ADS  and  Findings  document  available  to  the 
acquirer,  when  requested,  as  a  means  of  demonstrating  the  organization’s  achievements  in  process 
improvement.  Additional  information  about  the  ADS  is  provided  in  Section  3.6. 
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The  SCAMPI  Class  A  appraisal  is  described  in  detail  in  the  SCAMPI  Method  Definition 
Document  [SEI  2006b]. 

SCAMPI  Class  B  and  Class  C  appraisals  are  described  in  the  Handbook  for  Conducting  Standard 
CMMI  Appraisal  Method  for  Process  Improvement  (SCAMPI)  Class  B  and  C  Appraisals  [Hayes 
2005], 

1.3  INTERPRETING  CMMI  RATINGS 

Various  claims  made  about  capability  models  and  maturity  level  ratings  over  the  years  have  led  to 
some  misperceptions.  This  section  addresses  these  misperceptions.  Organizations  that  attain  a 
CMMI  maturity  level  rating  of  3  or  higher  should  have  documented  organizational  standards, 
defined  processes,  and  procedures  that  are  demonstrated  to  be  repeatable.  While  a  maturity  level  3 
supplier  should  tailor  the  organization’s  standard  processes  for  use  on  all  new  programs,  the 
acquirer  cannot  infer  that  the  processes  are  necessarily  effective  or  even  consistently  applied 
across  the  enterprise.  Some  warnings  for  the  acquirer  follow.  The  discussion  centers  on  maturity 
level  ratings  rather  than  process  capability  profiles,  since  the  majority  of  appraised  organizations 
end  up  with  a  maturity  level  rating,  and  maturity  levels  are  most  often  touted  by  suppliers. 

A  CMMI  rating  or  CMMI  level  is  not  a  guarantee  of  program  success.  While  process 
capability  and  organizational  maturity  have  been  shown  to  be  critical  factors  for  program  success, 
they  are  not  the  only  ones.  Mature  suppliers  that  have  capable  processes  in  place  on  a  program 
can  still  encounter  problems.  There  are  a  number  of  factors  that  contribute  to  program  success  or 
failure,  such  as  an  incomplete  understanding  of  the  job,  program  constraints  (e.g.,  funding, 
schedule),  the  skills  of  the  people  assigned  to  the  program,  and  the  maturity  of  the  technology 
applied  to  the  program.  Organizations  anticipate  that  adoption  of  CMMI-DEV  will  contribute  to 
program  success  because  it  is  an  improvement  over  previous  capability  maturity  models  and 
because  it  integrates  both  systems  and  software  engineering  in  a  single  framework  that  instills 
development  discipline.  However,  the  acquirer  cannot  assume  that  the  supplier  will  execute  the 
program  at  the  same  maturity  level  that  the  organization  has  achieved  and  advertised. 

Organizations  that  have  attained  CMMI  maturity  level  ratings  do  not  necessarily  apply 
those  appraised  processes  to  a  new  program  at  program  start-up.  Maturity  level  ratings  are 
achieved  by  an  organizational  unit  (e.g.,  the  corporation,  a  major  division,  a  few  key  programs) 
and  are  not  an  indication  of  how  an  individual  program  is  performing.  CMMI  maturity  level 
ratings  indicate  potential  organizational  performance  and  how  the  next  program  could  perform 
based  on  a  sampling  of  past  and  current  programs  if  the  appraised  processes  are  applied  in  the 
same  manner  as  on  the  chosen  sample  programs.  While  organizations  with  maturity  level  3  and 
higher  maturity  level  ratings  are  expected  to  employ  their  organizational  processes  on  new 
programs,  there  is  evidence  that  some  organizations  do  not  or  that  it  takes  some  time  before 
processes  are  applied  to  new  programs.  Therefore,  acquirers  should  determine  if  rated 
organizations  have  a  policy  for  using  their  organizational  processes  on  new  programs  and  a 
history  of  following  that  policy.  CMMI-DEV,  VI. 2  added  material  to  explicitly  address  process 
deployment  to  newly  started  programs. 

Organizations  that  claim  CMMI  ratings  are  not  always  dedicated  to  process  improvement. 

While  most  organizations  that  invest  in  the  improvement  of  their  development  processes  take  the 
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investment  seriously  and  commit  to  implementing  those  processes,  some  apparently  seek  a 
maturity  level  rating  only  for  use  in  pursuit  of  opportunities  (e.g.,  so  that  they  can  present  the 
credential  in  proposals).  These  organizations,  once  they  have  achieved  their  maturity  rating,  may 
not  apply  the  appraised  processes  as  implied. 

Additionally,  organizations  not  committed  to  process  improvement  may  choose  to  abandon  their 
processes  during  contract  execution  when  they  perceive  it  to  be  to  their  advantage  (e.g.,  costs  start 
to  overrun  or  schedule  slips  occur).  These  pressures  can  cause  the  acquirer  or  the  supplier  to 
abandon  mature  processes  by  not  performing  certain  design,  analysis,  or  verification  activities  in 
an  attempt  to  save  time  or  money.  Such  supplier  organizations  expose  acquirers  to  additional  risk 
through  poor  program  execution.  However,  there  are  ways  for  acquirers  to  identify  organizations 
that  give  short  shrift  to  process  improvement  and  are  not  committed  to  using  their  processes  as  a 
normal  part  of  their  work.  For  example,  organizations  that  are  actively  pursuing  process 
improvement  have  a  demonstrated  change  history  regarding  the  use  of  their  processes,  policies, 
and  procedures.  Acquirers  can  request  documentation  in  the  solicitation  that  provides  a  clear 
indicator  of  active  use,  monitoring,  and  improvement  of  these  artifacts.  Additionally,  suppliers 
may  use  other  tools,  complementary  to  CMMI,  that  provide  an  indication  of  their  commitment  to 
process  improvement,  such  as  Lean  Six  Sigma  or  Team  Software  ProcessSM  (TSPsm).4 

Organizations  may  sample  only  a  few  exemplar  programs  and  declare  that  all  programs  are 
being  executed  at  that  CMMI  level  rating.  Ratings  apply  to  an  organizational  unit;  however,  it 
is  often  not  practical  for  a  whole  organization  to  be  appraised.  Historically,  organizations  identify 
to  appraisal  teams  those  programs  that  are  available  to  be  reviewed  in  the  upcoming  appraisal. 

The  organization  can  be  expected  to  nominate  successful  programs  instead  of  those  having 
difficulties.  The  appraisal  team  selects  the  number  of  programs  that  it  can  reasonably  appraise  in 
the  allotted  time  and  that  it  deems  to  be  representative  programs  of  the  organizational  unit  from 
this  identified  set. 

Recent  changes  to  CMMI  are  intended  to  improve  program  selection  integrity  by  placing  more 
responsibility  on  Lead  Appraisers  to  ensure  a  representative  sample.  The  vl.2  ADS  includes 
information  about  the  type  and  number  of  programs  sampled  within  the  organization  and  what 
portion  of  the  development  population  the  appraised  programs  represent.  This  improvement  in  the 
ADS  seeks  to  prevent  “cherry-picking”  of  programs  and  provides  acquirers  with  a  better 
understanding  of  the  breadth  of  the  appraisal.  These  changes  are  not  reflected  in  earlier  versions 
of  the  ADS. 

Most  organizations  are  far  from  being  homogeneous  entities.  In  fact,  many  defense  suppliers  are 
an  amalgamation  of  smaller  organizations  acquired  over  time.  They  are,  therefore,  dispersed 
geographically  and  bring  different  legacy  processes  to  the  table.  Even  though  one  would  like  to 
believe  that  all  sectors,  divisions,  and  locations  of  a  supplier  operate  in  the  same  way,  this  is 
seldom  the  case.  Consequently,  one  supplier  location  may  indeed  utilize  verified,  capable 
processes,  while  another  location  may  not.  Thus,  awarding  a  contract  to  a  supplier  at  location  A 


4  For  more  information  on  continuous  process  improvement,  see  the  Continuous  Process  Improvement  Transfor¬ 
mation  Guidebook  [DoD  2006a], 
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that  has  capable  processes  at  location  B  is  not  a  guarantee  that  the  capable  processes  will  be 
applied  to  the  program  at  location  A  immediately,  or  ever. 

Organizations  that  claim  a  high  maturity  level  rating  (level  4  and  5)  are  not  necessarily 
better  suppliers  than  a  level  3  supplier.  Maturity  levels  4  and  5,  when  compared  across  different 
suppliers,  are  not  created  equal.  An  organization  must  select  one  process  or  subprocess  that  is 
important  to  its  business  objectives  and  quantitatively  manage  (maturity  level  4)  or  optimize 
(maturity  level  5)  it.  Another  organization  would  likely  select  different  processes  or  subprocesses. 
Thus,  if  the  chosen  processes  or  subprocesses  are  not  those  of  value  to  the  current  program,  the 
supplier’s  high  maturity  activity  may  not  add  benefit.  With  vl.2  of  the  ADS,  Lead  Appraisers 
have  the  additional  responsibility  of  verifying  that  the  chosen  processes  relate  to  the 
organization’s  business  objectives;  however,  acquisition  organizations  should  have  insight  into 
the  profile  of  capability  levels  across  all  the  process  areas  that  are  critical  or  relevant  to  the 
program.  Examination  of  the  capability  profile  clearly  shows  the  levels  of  any  particular  process 
area,  while  relying  on  a  maturity  level  will  not.  In  this  regard,  the  profile  is  more  meaningful  than 
relying  on  a  single  maturity  level. 

A  high  maturity  level  rating  is  not  an  indication  of  how  the  next  program  will  perform',  it  is  only 
an  indication  of  how  the  next  program  could  perform  provided  other  critical  success  factors 
remain  the  same.  Organizations  that  have  attained  high  maturity  level  ratings  are  not  immune  to 
the  aforementioned  pitfalls.  Achieving  a  high  maturity  rating  often  involves  modifying  the 
behavior  of  a  number  of  individuals  throughout  the  organization  and  disrupting  organizational 
inertia.  It  takes  management  focus  and  staff  commitment  to  improve.  Unfortunately,  there  is  a 
tendency  to  “take  a  breath”  after  the  organizational  surge  that  produced  the  maturity  level  rating. 
Management  must  be  diligent  to  assure  that  once  organizational  change  occurs,  the  behaviors 
become  institutionalized.  Clearly,  past  performance  only  indicates  the  potential  perfonnance  for 
the  current  program.  To  achieve  the  anticipated  performance  on  the  current  program,  the  mature 
organization  must  instantiate  the  appraised  processes  on  that  program. 

1.4  OVERVIEW  OF  THIS  GUIDEBOOK 

Although  many  organizations  have  adopted  CMMI,  nothing  in  the  rules  of  government 
acquisition  demands  the  use  of  CMMI  by  government  suppliers.  While  some  govermnent 
organizations  specified  that  their  suppliers  demonstrate  achievement  of  maturity  level  3  to  the 
Capability  Maturity  Model  for  Software  (SW-CMM  )  in  the  1990s,  and  some  continue  that 
practice  today  with  CMMI,  the  DoD  has  never  promulgated  a  policy  requiring  adherence  to  any 
CMMI  maturity  level  rating.  DoD  does  not  place  significant  emphasis  on  capability  level  or 
maturity  level  ratings,  but  rather  promotes  CMMI  as  a  tool  for  internal  process  improvement.  This 
guidebook  continues  that  theme. 

While  this  guidebook  is  intended  for  use  by  those  without  an  extensive  background  in  CMMI, 
some  readers  may  feel  that  they  need  additional  information  on  process  capability  and  process 
improvement.  Additional  sources  of  help  include  the  SEI,  the  Defense  Contract  Management 
Agency  (DCMA),  and  the  various  software  engineering  centers  (SECs)  or  software  support 
activities  (SSA)  established  within  the  Services. 
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If  acquirers  choose  to  capitalize  on  their  suppliers’  use  of  CMMI-DEV,  the  remainder  of  this 
guidebook  describes  approaches  for  solicitation  planning  and  execution  and  contract  monitoring: 

•  Chapter  2  covers  identifying  and  understanding  the  critical  process  areas  of  the  program. 

•  Chapter  3  describes  approaches  for  leveraging  the  process  capabilities  of  the  supplier  by 
enabling  them  in  the  request  for  proposal  (RFP). 

•  Chapter  4  describes  how  to  monitor  the  supplier’s  performance  to  ensure  that  the  identified 
process  capabilities  are  being  applied  to  the  program. 

The  implementation  of  these  approaches  is  discussed  in  detail  in  the  appendices.  The  acquirer 
must  determine  which  of  these  methods  are  applicable  to  its  program.  Appendix  A  contains  a 
checklist  of  the  methods  identified  in  this  document.  It  can  be  used  as  an  aid  in  implementing  and 
tracking  the  status  of  each  selected  method. 
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2  Identify  the  Critical  Process  Areas  of  the  Program 


Identifying  and  understanding  the  processes  that  are  most  critical  to  the  success  of  the  program  is 
an  important  first  step  in  determining  the  criticality  of  process  related  risks.  This  chapter  describes 
how  to  determine  the  critical  process  areas  of  the  program.  The  more  critical  a  process  may  be  to 
the  program’s  success,  the  more  importance  should  be  placed  on  the  capability  of  the  supplier’s 
process,  and  the  more  risk  may  be  incurred  if  its  process  is  not  well  defined  and  implemented. 

Suppliers  that  establish  and  use  effective  processes,  working  as  part  of  a  mature  organization,  are 
more  likely  to  do  the  following: 

•  plan  and  execute  development  programs  predictably 

•  support  those  programs  with  an  engineering  and  management  infrastructure  that  will  enable 
them  to  succeed 

•  communicate  more  accurately  and  frequently  with  customers  and  stakeholders 

•  stay  on  schedule  and  deliver  quality  products 

Evaluation  of  the  supplier’s  proposed  development  processes — and  then  ensuring  that  the  supplier 
actually  applies  those  processes  to  the  program  from  the  start — are  the  acquirer’s  primary  process 
challenges.  To  meet  these  challenges,  the  acquirer  must  first  understand  the  critical  process  areas 
of  the  program. 

2.1  ESTIMATE  PROGRAM  DEPENDENCE  ON  PROCESS  CAPABILITY 

Experience  has  shown  that  the  program  attributes  in  Table  2  may  make  a  program  more 
vulnerable  to  immature  development  processes.  These  six  attributes  are  a  critical  subset  of  risk- 
inducing  program  characteristics.  Other  attributes  of  the  program  may  also  indicate  vulnerabilities 
to  immature  supplier  development  processes. 
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Table  2:  Factors  Driving  the  Need  for  Process  Capability 


Program  Attribute 

Discussion 

Effect  of  Immature  Development  Processes 

Complexity 

This  attribute  includes  factors  such 
as  the  number  and  complexity  of 
interfaces,  integration  complexity,  the 
number  and  diversity  of  stakeholders, 
software  reuse,  the  degree  of 
interoperability  with  other  systems, 
and  domain  complexity. 

Immature  development  processes  reduce  the 
program’s  ability  to  deal  with  the  planning, 
execution,  and  oversight  of  complex 
programs. 

Size 

Program  size  indicators  include  the 
number  of  developmental 
components,  the  number  of  acquirer 
and  supplier  teams  participating  in 
the  development  lifecycle,  and  the 
number  of  user,  system,  or  software 
requirements. 

Large  programs  require  geometrically 
increasing  lines  of  communication  for 
oversight  and  stakeholder  involvement  in 
various  lifecycle  development  activities. 
Immature  processes  typically  result  in 
inaccurate  or  late  communication,  poor 
planning,  a  lack  of  teamwork,  and  rework. 

Lack  of  precedent 

This  attribute  includes  five  aspects  of 

the  program: 

1 .  Has  this  system  functionality 
been  developed  before  with 
these  quality  attributes? 

2.  How  mature  is  the  technology? 

3.  Has  the  target  technology  been 
applied  in  this  domain  before? 

4.  Does  this  team  have  experience 
with  this  type  of  system? 

5.  Are  disparate  systems  or 
technologies  brought  together  to 
provide  a  new  integrated 
capability? 

Immature  processes  can  fail  under  the 
pressure  of  responding  to  unprecedented 
challenges.  Mature  programs  with  highly 
disciplined  team  members  have  a  better 
chance  of  rising  to  the  challenge. 

Dependence  on  software 

This  attribute  can  generally  be 
described  by  its  components: 
proportion  and  reliance.  Proportion 
reflects  the  relative  investment  in 
software  development  versus 
hardware  or  systems  development. 
Reliance  indicates  functional 
dependence  of  the  system  on  the 
software. 

Development  of  a  system  that  is  software 
dependent  imposes  challenges  at  the  system, 
software,  and,  potentially,  hardware  levels. 
Immature  processes  may  result  in  a 
breakdown  of  basic  engineering  functions, 
such  as  requirements  allocation,  the 
development  of  coherent  architectures,  and 
system  integration  and  testing. 

Schedule  or  funding 
constraints 

This  attribute  reflects  program 
imperatives  to  deliver  a  useful 
capability  to  the  customer  for  a 
limited  expenditure.  “Hard”  delivery 
dates  and  inadequate  funds  stress 
both  the  acquirer  and  the  supplier 
team. 

Programs  that  exhibit  immature  processes 
have  little  or  no  capability  to  react  reasonably 
to  schedule  or  funding  stress,  to  properly 
manage  expectations,  or  to  effectively 
manage  the  resulting  risks. 

Requirements  instability 

There  is  always  some  level  of 
requirements  evolution  for  complex, 
unprecedented  systems.  However,  if 
user  needs  or  system  requirements 
are  highly  volatile,  either  through  lack 
of  definition  or  in  response  to 
changing  customer  needs,  excessive 
requirements  variation  will  occur  at 
the  system  and  software  levels. 

Programs  with  immature  processes  often  find 
it  difficult  to  respond  to  a  changing 
requirements  base  from  either  an  engineering 
or  management  perspective. 

Table  3  is  a  diagnostic  tool  that  the  acquirer  can  use  to  judge  the  program’s  rating  with  respect  to 
six  attributes.  For  each  of  the  attributes  shown  in  the  left  column,  the  program  is  rated  low, 
medium,  or  high.  For  example,  if  schedule  urgency  is  driving  the  program  to  be  kicked  off  before 
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user  needs  are  clearly  understood,  both  Schedule  and  Funding  Constraints  and  Requirements 
Instability  might  be  rated  high. 


Table  3:  Program  Characterization 


Program  Attribute 

Ratings 

Low 

Medium 

High 

Complexity 

Size 

Lack  of  Precedent 

Dependence  on  Software 

Schedule  or  Funding  Constraints 

Requirements  Instability 

TOTAL 

The  results  of  this  diagnostic  may  be  interpreted  as  follows: 

•  If  the  program  has  no  high  ratings  and  no  more  than  two  medium  ratings,  it  probably  has  a 
low  dependence  on  process  capability. 

•  If  the  program  has  one  high  rating  or  three  or  more  medium  ratings,  it  probably  has  a 
moderate  dependence  on  process  capability. 

•  If  the  program  has  two  or  more  high  risk  ratings,  it  probably  has  a  high  dependence  on 
process  capability. 

If  the  program  is  rated  as  moderate  or  high,  the  process  capabilities  of  the  suppliers  as  they  are 
applied  during  the  development  effort  can  be  a  critical  factor  in  the  success  of  the  program,  and 
the  approaches  in  this  guidebook  may  be  used  to  alleviate  some  of  the  risk. 

2.2  UNDERSTAND  THE  RELEVANT  PROCESS  AREAS 

In  nearly  every  program,  all  17  of  the  CMMI-DEV  process  areas  related  to  Project  Management, 
Engineering,  and  Support  are  used  at  some  time  during  the  program’s  lifecycle.  Of  the  process 
areas  included  at  maturity  level  2  and  3  related  directly  to  the  execution  of  the  program,  some  are 
more  important  than  others  at  different  phases  in  the  program  lifecycle.  The  capability  of  these 
processes  at  the  time  when  they  are  most  needed  determines  the  risk  that  they  pose  to  the 
program.  While  all  process  areas  must  be  planned  for  early  in  the  lifecycle  and  can  apply 
throughout  the  lifecycle,  it  may  be  beneficial  for  some  to  begin  earlier  in  the  program  than  others, 
as  illustrated  in  Figure  3: 
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Figure  3:  Notional  Time  Sequencing  of  Process  Areas  Mapped  to  Development  Lifecycle 

The  process  areas  of  interest  and  their  corresponding  purpose  statements  are  as  follows: 

•  Configuration  Management  -  to  establish  and  maintain  the  integrity  of  work  products  using 
configuration  identification,  configuration  control,  configuration  status  accounting,  and 
configuration  audits. 

•  Decision  Analysis  and  Resolution  —  to  analyze  possible  decisions  using  a  formal  evaluation 
process  that  evaluates  identified  alternatives  against  established  criteria. 

•  Integrated  Project  Management  -  to  establish  and  manage  the  project  and  the  involvement  of 
the  relevant  stakeholders  according  to  an  integrated  and  defined  process  that  is  tailored  from 
the  organization’s  set  of  standard  processes. 

•  Measurement  and  Analysis  -  to  develop  and  sustain  a  measurement  capability  that  is  used  to 
support  management  information  needs. 

•  Process  and  Product  Quality  Assurance  -  to  provide  staff  and  management  with  objective 
insight  into  processes  and  associated  work  products. 

•  Product  Integration  -  to  assemble  the  product  from  the  product  components,  ensure  that  the 
product,  as  integrated,  functions  properly,  and  deliver  the  product. 

•  Project  Monitoring  and  Control  -  to  provide  an  understanding  of  the  project’s  progress  so 
that  appropriate  corrective  actions  can  be  taken  when  the  project’s  performance  deviates 
significantly  from  the  plan. 

•  Project  Planning  -  to  establish  and  maintain  plans  that  define  project  activities. 

•  Requirements  Development  -  to  produce  and  analyze  customer,  product,  and  product 
component  requirements. 
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•  Requirements  Management  —  to  manage  the  requirements  of  the  project’s  products  and 
product  components  and  to  identify  inconsistencies  between  those  requirements  and  the 
project’s  plans  and  work  products. 

•  Risk  Management  -  to  identify  potential  problems  before  they  occur  so  that  risk-handling 
activities  can  be  planned  and  invoked  as  needed  across  the  life  of  the  product  or  project  to 
mitigate  adverse  impacts  on  achieving  objectives. 

•  Supplier  Agreement  Management  -  to  manage  the  acquisition  of  products  from  suppliers. 

•  Technical  Solution -to  design,  develop,  and  implement  solutions  to  requirements.  Solutions, 
designs,  and  implementations  encompass  products,  product  components,  and  product-related 
lifecycle  processes  either  singly  or  in  combination  as  appropriate. 

•  Validation  -  to  demonstrate  that  a  product  or  product  component  fulfills  its  intended  use 
when  placed  in  its  intended  environment. 

•  Verification  -  to  ensure  that  selected  work  products  meet  their  specified  requirements. 

These  process  areas  align  well  with  the  technical  and  technical  management  processes  described 
in  Section  4.2  of  the  Defense  Acquisition  Guidebook  (DAG),  Chapter  4,  “Systems  Engineering” 
[DoD  2004].  (See  Appendix  E  on  page  67  for  more  information  about  mapping  between  DAG 
Chapter  4  processes  and  CMMI  process  areas.)  According  to  Section  4.5.1  of  the  DAG,  the 
systems  engineering  plan  (SEP)  will  include  the  following: 

The  systems  engineering  processes  to  be  applied  in  the  program  (e.g.,from  a 
standard,  a  capability  maturity  model,  or  the  supplier's  process).  Describe  how  the 
processes  will  be  implemented  and  how  they  will  be  tailored  to  meet  individual 
acquisition  phase  objectives.  Describe  how  the  systems  engineering  processes  will 
support  the  technical  and  programmatic  products  required  of  each  phase.  Sections 
4.2  (process)  and  4.3  (process  application  to  SE  phase)  provide  a  “roadmap”  of  how 
SE  processes  can  be  applied  to  an  acquisition  program. 


2.3  ASSESS  AND  PRIORITIZE  THE  CRITICAL  PROCESS  AREAS  FOR  THE  PROGRAM 

Based  on  the  previous  discussion  and  the  context  of  the  envisioned  program,  the  acquirer  can 
determine  which  process  areas  are  critical.  Two  methods  are  described  in  this  section  that  can  be 
used  to  assist  the  acquirer  with  this  determination.  The  objective  is  to  assess  and  prioritize  the 
critical  process  areas  of  the  program,  using  CMMI-DEV  as  the  reference  model.  Once  identified, 
the  program’s  critical  process  areas  can  be  addressed  in  the  RFP.  Supplier  processes  that  relate  to 
those  critical  areas  can  then  be  evaluated  in  relationship  to  the  risk  that  poorly  defined  or  executed 
processes  can  create. 

2.3.1  Self  Assessment 

A  self-assessment  of  the  program’s  critical  process  areas  can  be  performed  as  part  of  acquisition 
strategy  development.  Knowing  the  critical  process  areas  helps  to  identify  the  process  capabilities 
that  a  successful  supplier  must  possess.  This  method  requires  that  the  acquirer  has  basic  process 
knowledge.  This  knowledge  should  be  resident  within  the  acquirer’s  office  under  nonnal 
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circumstances.  (See  “Self-Assessment  Questionnaire”  on  page  33  in  Appendix  B  for 
implementation  details.) 

2.3.2  Facilitated  Assessment 

A  facilitated  assessment  of  the  program’s  critical  process  areas  can  be  used  as  an  alternative  to  the 
self-assessment  method.  A  facilitated  assessment  may  be  performed  by  a  trained  and  qualified 
outside  agency  with  both  domain  and  process  expertise.  This  method  requires  sufficient  funding 
and  time  for  external  support.  (See  “What  to  Do”  in  the  “Self-Assessment  Questionnaire”  on  page 
36  section  in  Appendix  B  for  implementation  details.) 
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3  Leverage  the  Process  Capabilities  of  the  Supplier 


Chapter  2  described  how  to  identify  process  areas  that  are  critical  to  a  program  and  determine 
what  to  concentrate  on  when  considering  the  supplier’s  approaches.  This  chapter  includes 
methods  for  gathering  information  about  the  capabilities  of  suppliers  to  perform  development 
activities.  The  acquirer  can  use  these  methods  to  determine  organizational  strengths,  weaknesses, 
or  risks  associated  with  a  supplier’s  approach  and  organizational  processes.  These  methods  are 
fully  independent  and  may  be  used  individually  or  in  combination.  Each  method  is  useful  in 
limiting  process  performance  risk  within  an  acquisition,  but  the  acquirer  can  balance  the  need  for 
information  with  the  resources  available  to  review  that  information.  Table  4,  at  the  end  of  the 
chapter,  provides  guidance  on  selecting  the  methods  that  may  be  most  appropriate  for  the 
acquisition  program. 

Once  the  acquirer  selects  one  or  more  of  these  methods,  they  have  to  be  enabled  in  an  RFP,  the 
contract,  or  in  some  other  manner.  Examples  of  sample  evaluation  criteria  and  instructions  that 
apply  to  Sections  M  and  L  of  the  RFP,  are  available  in  the  Guide  for  Integrating  Systems 
Engineering  into  DoD  Acquisition  Contracts  [DoD  2006f].  The  sample  language  in  that  guide 
should  be  adapted  to  align  with  specific  selections  of  the  following  methods. 

3.1  CONSIDER  PROCESS  PROPOSALS  IN  CRITICAL  AREAS 

One  way  for  the  acquirer  to  understand  the  supplier’s  plans  for  process  application  to  the  program 
is  to  request  and  evaluate  process  proposals  in  areas  critical  to  the  program.  When  preparing  a 
proposal,  the  offeror5  plans  and  estimates  the  work  that  will  be  done  on  the  program.  This 
estimate  is  based  on  the  offeror’s  intended  methods  for  executing  the  work  (i.e.,  its  work 
processes).  The  acquirer  can  ask  for  a  detailed  description  of  processes  in  areas  especially  critical 
to  program  success,  as  identified  in  Chapter  2.  Each  offeror’s  response  to  this  request  illustrates 
how  its  processes  are  adapted  to  and  applied  to  the  program. 

Offerors  with  a  strong  process  focus  and  capable  processes  (i.e.,  maturity  level  3  or  capability 
level  3  as  defined  by  CMMI-DEV)  already  have  process  descriptions.  Offerors  who  have  not 
implemented  CMMI-DEV  can  still  participate  by  submitting  the  specific  processes  that  they  plan 
to  use  on  the  program. 

The  acquirer  must  balance  the  desire  for  detailed  information  against  the  impact  that  a  large 
volume  of  information  will  have  on  the  evaluation  effort.  Concentrating  on  only  the  most  critical 
process  areas,  as  defined  in  Chapter  2,  may  help  achieve  such  a  balance. 

The  information  supplied  by  the  offeror  in  process  proposals  that  address  the  most  critical  process 
areas  can  be  formally  evaluated  as  part  of  the  source  selection.  The  SCAMPI  Class  C  process 
appraisal  method  can  be  used  to  review  the  process  information  from  the  proposals  such  as 
process  definitions,  templates,  process  descriptions,  and  sample  work  products.  The  appraisal 


5  An  offeror  is  a  supplier  competing  for  a  contract. 
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team  uses  this  process  information  and  the  practices  in  CMMI-DEV  as  a  reference  to  identify 
strengths  and  weaknesses  in  the  offeror’s  proposed  processes. 

Furthermore,  certain  aspects  of  the  proposed  processes  can  then  be  captured  in  the  contract, 
creating  an  obligation  for  the  chosen  supplier  to  implement  them  and  providing  a  basis  for 
contract  monitoring.  (See  “Consider  Process  Proposals  in  Critical  Areas”  on  page  40  in  Appendix 
C  for  implementation  details.) 

3.2  CONSIDER  INTEGRATED  MASTER  PLAN  DOCUMENTATION  OF  PROPOSED 
PROCESSES 

An  integrated  master  plan  (IMP)  that  describes  how  the  offeror  intends  to  execute  the  program  is 
linked  to  the  offeror’s  integrated  master  schedule  (IMS)  and  uses  the  same  program  work 
breakdown  structure  (WBS)  used  to  prepare  the  basis  of  estimate  (BoE)  documentation.  If  the 
proposed  processes  are  included  in  the  IMP,  the  acquirer  will  have  the  means  to  verify  process 
execution  without  additional  process  monitoring  actions. 

The  work  described  in  this  set  of  documentation  represents  how  each  offeror  has  planned  to 
execute  the  program  and,  as  such,  represents  a  commitment  to  the  incorporation  of  its  work 
processes  into  program  execution.  The  value  of  the  complete  package  is  that  it  enables  the 
acquirer  to  review  each  offeror’s  approach  to  implementing  the  program  in  a  holistic  sense, 
especially  if  the  offeror  is  asked  to  map  its  proposed  activities  to  its  processes  and  the  critical 
process  areas  identified  by  the  program.  This  mapping  can  be  done  with  little  additional 
documentation  and  allows  the  evaluator  to  see  how  well  the  offeror’s  processes  will  incorporate 
into  the  program  plan.  More  importantly,  the  acquirer  can  use  it  to  evaluate  the  degree  to  which 
each  offeror  intends  to  invoke  its  processes  and  practices,  especially  if  the  offeror  is  specifically 
asked  to  provide  that  information  in  the  IMP  as  well  as  provide  a  map  of  its  process  set  to  the 
program’s  identified  critical  process  areas  in  the  IMP  or  elsewhere  in  the  proposal.  This 
evaluation  provides  a  valuable  assessment  of  the  degree  to  which  each  offeror  can  be  expected  to 
execute  its  processes  after  contract  award  and  allows  the  evaluator  to  assign  an  appropriate  risk  to 
each  offeror.  (See  “Consider  Integrated  Master  Plan  Documentation  of  Proposed  Processes”  on 
page  42  in  Appendix  C  for  implementation  details.) 

General  guidance  on  the  IMP  and  IMS  can  be  found  in  the  Integrated  Master  Plan  and  Integrated 
Master  Schedule  Preparation  and  Use  Guide  [DoD  2005]. 

3.3  CONSIDER  INCORPORATION  OF  PROCESS  REVIEWS  INTO  PROPOSED 
PROGRAM  SCHEDULES 

Organizations  that  have  institutionalized  their  processes  should  conduct  periodic  reviews  or  audits 
of  their  processes.  Incorporation  of  these  process  reviews  in  the  offeror’s  IMS  and  IMP  provides 
visibility  into  the  rollout  of  the  offeror’s  processes  and  work  products  for  critical  program 
processes. 

The  acquirer  can  request  that  the  offeror  include  its  planned  process  reviews  in  its  IMP  and  IMS, 
including  a  list  of  process  work  products  for  the  program’s  critical  process  areas,  which  will  be 
included  in  the  earned  value  management  system  (EVMS)  and  initial  baseline  review  (IBR).  (See 
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“Consider  Incorporation  of  Process  Reviews  into  Proposed  Program  Schedules”  on  page  43  in 
Appendix  C  for  implementation  details.) 

3.4  CONSIDER  THE  APPROACH  TO  INTEGRATION  WITH  SUBCONTRACTOR 
PROCESSES 

Process  relationships  between  the  prime  contractor  (supplier)  and  major  development 
subcontractors  can  be  a  potential  source  of  risk  to  the  program.  Therefore,  it  is  important  for  the 
acquirer  to  know  and  understand  the  risks  arising  from  the  integration  of  prime  contractor  and 
subcontractor  processes  to  properly  monitor  and  manage  these  risks.  The  RFP  can  request  a  plan 
from  the  prime  contractor  addressing  this  subject,  and  it  can  ensure  that  all  activities  and  risks  are 
captured  in  the  IMP,  IMS,  and  risk  management  plan  (RMP).  (See  “Consider  Approach  to 
Integration  with  Subcontractor  Processes”  on  page  44  in  Appendix  C  for  implementation  details.) 

3.5  CONSIDER  THE  APPROACH  TO  INTEGRATION  WITH  ACQUIRER  PROCESSES 

Relationships  between  the  acquirer  and  supplier  processes  can  be  a  source  of  risk  to  the  program. 
Since  the  acquirer  and  the  supplier  have  significantly  different  roles  and  responsibilities  in  an 
acquisition  program,  their  processes  differ  significantly.  However,  in  areas  where  responsibility 
for  processes  is  shared  or  where  significant  interaction  exists  between  the  acquirer  and  supplier 
processes,  it  is  desirable  to  ensure  that  supplier  processes  critical  to  the  program  (identified  in 
Chapter  2)  are  compatible  with  the  acquirer  processes.  Process  compatibility  requires  the 
following: 

•  A  common  vocabulary  (e.g.,  what  is  the  meaning  of  a  high-risk  exposure) 

•  Consistent  and  compatible  process  objectives 

•  Efficient  and  clear  bidirectional  communication  of  process  performance 

•  A  clear  understanding  of  the  acquirer  and  supplier  processes 

•  Alignment  of  supplier  lifecycle  phase  deliverables  and  work  products  with  the  acquirer’s 
lifecycle  expectations 

Examples  of  process  integration  include  the  following: 

•  The  acquirer  and  the  supplier  share  responsibility  for  risk  management.  To  ensure  effective 
risk  management  for  the  program,  common  definitions  can  be  established  for  risk  impact, 
risk  probability,  etc.  Furthermore,  guidelines  can  be  established  to  help  determine  which 
risks  are  the  responsibility  of  the  supplier,  which  are  the  responsibility  of  the  acquirer,  and 
which  are  shared.  Issues  such  as  the  frequency  of  risk  reporting  must  also  be  addressed.6 

•  Both  the  acquirer  and  the  supplier  will  perform  requirements  management  and  requirements 
development  activities.  The  acquirer  will  translate  stakeholder  needs  (e.g.,  user 
requirements)  into  contract  requirements.  These  requirements  must  be  accurate,  complete, 
and  consistent;  they  must  also  be  baselined  and  managed.  The  supplier  takes  the  contract 
requirements  and  derives  product-based  requirements  from  them.  These  requirements  must 


For  further  information  on  the  alignment  of  acquirer  and  supplier  risk  management  activities,  see  the  Risk  Man¬ 
agement  Guide  for  DoD  Acquisition  [DoD  2006e], 
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also  be  baselined  and  managed.  To  ensure  satisfaction  of  user  needs,  traceability  must  be 
maintained  from  the  user  needs  through  the  product-based  requirements. 

•  Configuration  management  (CM)  is  a  shared  responsibility  between  the  acquirer  and  the 
supplier;  both  must  manage  the  configuration  of  program  documents  to  ensure  accurate 
communication  and  real-time  understanding  of  program  status.  Coordination  of  CM 
processes  creates  a  consistent  baseline  on  which  both  the  acquirer  and  the  supplier  can  base 
their  efforts. 

The  acquirer  can  monitor  and  manage  any  risks  arising  from  the  integration  of  its  processes  and 
the  supplier  processes  once  they  are  known.  (See  “Consider  Approach  to  Integration  with 
Acquirer  Processes”  on  page  45  in  Appendix  C  for  implementation  details.) 

3.6  CONSIDER  RECENT  APPRAISAL  RESULTS 

Many  organizations  perform  formal  process  appraisals  as  a  means  of  understanding  their  process 
capabilities  and  degree  of  compliance  with  CMMI.  In  these  appraisals,  a  representative  sampling 
of  programs  within  the  organization  is  examined  to  identify  evidence  that  best  practices  as  defined 
in  CMMI-DEV  were  deployed. 

Results  of  previous  organizational  appraisals  (within  the  past  3  years)  can  be  requested  and 
collected  from  each  prime  contractor  and  each  subcontractor  that  will  perform  major  development 
activities.  Knowledge  and  understanding  of  appraisal  results  can  help  the  acquirer  identify  the 
process  capabilities  of  the  supplier  in  relationship  to  the  critical  process  areas  identified  in 
Chapter  2. 

The  acquirer  should  look  for  answers  to  the  following  questions  from  data  supplied  on  the  ADS: 

•  Is  the  organization  named  in  the  ADS  the  same  organization  that  is  bidding  on  and  will 
execute  the  program? 

•  Was  the  bidding  organization  involved  in  the  appraisal? 

•  Was  the  appraisal  performed  by  a  third  party  not  affiliated  with  the  organization  or  the 

company,  or  by  a  Lead  Appraiser  associated  with  the  appraised  organization? 

•  How  many  programs  were  included  in  the  appraisal? 

•  Was  the  appraisal  performed  less  than  two  years  ago? 

•  What  was  the  appraisal  method  used? 

•  Which  process  areas  were  included  in  the  appraisal? 

•  What  is  the  organizational  significance  of  the  appraisal  sponsor? 

Additional  questions  that  can  be  asked  about  the  appraisal  findings  include  the  following: 

•  Do  the  appraisal  findings  match  the  scope  of  the  ADS  or  do  they  document  a  modification  of 
the  appraisal  scope? 

•  Do  the  appraisal  findings  declare  any  process  areas  not  applicable  (NA)? 

•  Will  weaknesses  noted  in  the  appraisal  findings  have  a  significant  impact  on  the  program? 

(See  “Consider  Recent  Appraisal  Results”  on  page  46  in  Appendix  C  for  implementation  details.) 
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3.7  CONSIDER  HISTORICAL  DATA  ON  PROCESS  PERFORMANCE 


Organizations  that  have  institutionalized  their  processes  should  collect  and  maintain  data  on  the 
historical  use  of  their  processes  on  programs.  Data  may  include  statistics  on  the  following: 

•  Breadth  of  process  application  across  the  organization  (e.g.,  what  percentage  of  programs 
employ  standard  or  tailored  versions  of  the  organization’s  standard  processes) 

•  Consistency  of  process  application  (e.g.,  what  percentage  of  processes  applied  to  programs 
are  modified  from  the  organization’s  standard  processes) 

•  Program  compliance  with  the  defined  processes  (e.g.,  metrics  resulting  from  process 
monitoring  or  process  audits) 

The  acquirer  may  request  and  evaluate  any  relevant  data  on  historical  process  performance.  As 
part  of  the  evaluation  of  the  past  performance  of  source  selection,  the  acquirer  may  ask  the  offeror 
to  identify  the  historical  performance  of  selected  processes  for  the  number  of  completed  programs 
the  offeror  submitted  for  consideration.  In  addition,  the  acquirer  may  request  that  past 
performance  data  be  submitted  for  on-going  programs  coincident  with  the  present  bid, 
recognizing  that  some  of  these  programs  may  not  be  sufficiently  advanced  in  their  lifecycle  to 
have  completion  data  available.  The  acquirer  may  also  contact  DCMA  to  obtain  past  performance 
assessment  information  on  suppliers.7 8  (See  “Consider  Historical  Data  on  Process  Performance” 
on  page  5 1  in  Appendix  C  for  implementation  details.) 

3.8  CONSIDER  RECENT  POST-AWARD  APPRAISAL  DATA  FROM  OTHER  PROGRAMS 

Over  the  past  few  years,  several  government  acquisition  programs  planned  for  and  executed  post¬ 
award  appraisals  within  the  winning  supplier  or  supplier  team.  These  appraisals  were  performed 
by  trained  and  authorized  appraisal  teams  consisting  of  members  from  the  acquisition 
organization,  the  supplier  organizations,  and  independent  members.  The  purpose  of  these 
appraisals  was  to  address  the  process-related  risks  of  the  appraised  program.  These  appraisals  are 
limited  to  the  scope  of  the  program  and  the  portion  of  the  organization  executing  the  program. 

The  purpose  is  not  to  evaluate  an  organization’s  process  capability,  but  to  identify  risks 
introduced  into  the  program  by  process  performance. 

These  appraisals  have  been  performed  at  several  stages  of  the  program  lifecycle  and,  therefore, 
may  or  may  not  be  useful  to  the  program: 

•  Some  were  performed  near  the  end  of  one  phase  of  system  development,  preceding  a  down- 

i  8 

select  to  provide  input  to  the  source  selection  process. 

•  Some  were  performed  shortly  after  contract  award  to  encourage  the  supplier  to  instantiate  the 
necessary  processes  on  the  program  quickly. 


7  See  also  the  Guide  for  Integrating  Systems  Engineering  into  DoD  Acquisition  Contracts  [DoD  2006f|. 

8  A  downselect  is  an  acquisition  term  that  describes  the  process  of  determining  a  winning  bid  after  two  or  more 
competing  contractors  demonstrate  their  designs  or  prototypes.  Often,  to  alleviate  program  risk,  an  acquisition 
strategy  will  award  two  contracts  to  two  offeror  teams  with  competing  designs.  Downselect  is  the  term  used  when 
one  design  or  prototype  is  finally  selected  after  it  is  proven  to  be  superior  in  nature  to  the  others. 
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•  Some  were  performed  later  in  the  development  lifecycle  to  monitor  the  supplier’s 
conformance  to  process  performance  commitments. 

(See  “Consider  Recent  Post- A  ward  Appraisal  Data  from  Other  Programs”  on  page  53  in 
Appendix  C  for  implementation  details.) 

3.9  CONSIDER  PROCESS  COMPLIANCE  EVALUATIONS  DURING  EXECUTION 

The  degree  to  which  a  program  complies  with  the  organization’s  set  of  standard  processes  is 
extremely  important  and  indicates  how  the  offeror  might  perform  on  the  program.  The 
compliance  of  the  offeror’s  standard  processes  should  be  measured  early  in  each  program’s 
lifecycle  and  remeasured  at  reasonable  intervals  throughout  the  lifecycle. 

During  the  execution  of  the  program,  the  acquirer  can  monitor  the  implemented  processes  to 
ensure  the  program  is  benefiting  from  the  process  capabilities  of  the  supplier.  There  are  several 
ways  to  monitor  a  supplier’s  process  performance: 

•  If  organizations  include  process  compliance  evaluations  as  part  of  regular  quality  assurance 
reviews,  the  acquirer  should  obtain  and  review  their  collected  data. 

•  If  organizations  evaluate  process  performance  as  part  of  a  global  mission  assurance  review 
process,  the  acquirer  should  obtain  and  review  the  collected  data. 

•  The  acquirer  should  contact  the  local  DCMA  office  to  request  performance  of  independent 
process  reviews. 

(See  “Consider  Process  Compliance  Evaluations  during  Execution”  on  page  53  in  Appendix  C  for 
implementation  details.) 

3.10  CONSIDER  THE  REPORTING  OF  PROCESS  STATISTICS 

Receiving  reports  of  process  statistics  can  provide  the  acquirer  with  near-continuous  insight  into 
the  supplier’s  process  capability.  Organizations  that  have  institutionalized  their  processes,  monitor 
and  control  them.  As  a  fundamental  part  of  the  supplier’s  own  process  monitoring  activities,  it 
collects,  analyzes,  and  uses  process  metrics  to  ensure  its  processes  are  being  applied,  are 
appropriate,  and  are  being  used  as  a  basis  for  improvement.  As  part  of  the  metrics  program,  the 
acquirer  can  include  reporting  of  process  performance  metrics  (e.g.,  process  compliance  data  or 
defect  rate  data)  in  the  RFP.  Ensure  that  this  reporting  requirement  is  incorporated  into  the 
contract.  (See  “Consider  the  Reporting  of  Process  Statistics”  on  page  54  in  Appendix  C  for 
implementation  details.) 

3.11  CONSIDER  ACCESS  TO  SELECTED  PROCESS  ARTIFACTS 

Executing  a  process  results  in  work  products,  also  called  process  artifacts.  Examining  these  work 
products  can  provide  insight  into  process  performance.  For  example,  if  a  peer  review  process  calls 
for  the  publication  of  meeting  minutes,  the  presence  (or  absence)  of  those  minutes  can  give  some 
idea  of  the  number  of  peer  reviews,  the  subjects  of  peer  reviews,  and  even  the  quality  of  the  peer 
reviews.  Likewise,  the  acquirer  can  evaluate  the  risk  management  process  by  periodically 
examining  the  risk  database  to  reveal  information  such  as  the  following: 
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•  Currency  of  the  program  risks:  Are  risks  being  added  and  resolved  on  an  ongoing  basis? 

•  Breadth  of  the  risk  management  process  deployment:  Who  is  identifying  and  managing 
risks? 

Many  process  artifacts  mentioned  in  CMMI-DEV  are  plans  or  other  documents  that  are  typically 
part  of  any  development  program.  The  acquirer  can  decide  which  of  the  artifacts  should  be 
delivered  formally  and  which  should  only  require  infonnal  access.  (See  “Consider  Access  to 
Selected  Process  Artifacts”  on  page  55  in  Appendix  C  for  implementation  details.) 

3.12  CONSIDER  CONDUCTING  PRE-AWARD  PROCESS  APPRAISALS 

While  the  results  of  previous  appraisals  using  CMMI-DEV  can  be  helpful  in  identifying  the 
supplier’s  process  capabilities,  they  may  not  be  sufficient  for  a  pending  program’s  needs  because 
of  the  following: 

•  The  portion  of  the  organization  bidding  on  this  program  was  not  involved  in  the  previous 
appraisal. 

•  The  programs  evaluated  in  the  previous  appraisal  were  not  in  the  same  technology  domain  as 
this  program. 

•  Processes  critical  to  this  program  were  not  appraised  in  the  previous  appraisal. 

If  sufficient  information  on  process  capability  has  not  been  submitted  by  the  offeror  to  make  the 
acquirer  comfortable  in  the  source  selection,  it  may  be  helpful  to  conduct  an  appraisal  of  the 
offeror’s  organizations  and  locations  performing  the  work  on  the  program.  Such  an  appraisal  can 
be  conducted  prior  to  the  release  of  the  RFP  in  an  Advisory  Multi-Step  Process9  or  after  the  RFP 
release  as  part  of  the  evaluation  of  the  source  selection.  While  it  is  generally  not  practical  to 
conduct  a  comprehensive  appraisal  of  all  process  areas  within  the  organization,  it  is  feasible  to 
conduct  an  appraisal  of  a  few  process  areas  as  a  means  of  filling  gaps  in  understanding  the 
offeror’s  process  capabilities.  A  SCAMPI  Class  B  or  Class  C  appraisal  may  be  used. 

An  appraisal  conducted  prior  to  or  during  source  selection  will  not  evaluate  the  specific  team  and 
the  specific  tailored  processes  that  will  execute  the  program.  Although  CMMI-DEV  requires  the 
early  deployment  of  processes  on  new  programs,  in  most  cases  the  program  team  will  not  exist 
and  the  processes  will  not  be  implemented  until  after  contract  award.  These  process  appraisals 
only  allow  the  acquirer  to  evaluate  the  process  capabilities  of  teams  similar  to  the  envisioned 
program  team,  working  on  a  similar  program  in  the  same  organization  and  environment  that  the 
actual  program  will  have.  (See  “Consider  Conducting  Pre-Award  Process  Appraisals”  on  page  56 
in  Appendix  C  for  implementation  details.) 

3.13  CONSIDER  CONDUCTING  POST-AWARD  PROCESS  APPRAISALS 

If  the  program’s  dependence  on  process  capability  was  determined  to  be  significant  using  the 
methods  prescribed  in  Chapter  2,  the  acquirer  can  require  that  the  winning  supplier  undergo  one 
or  more  process  appraisals  of  the  program’s  critical  process  areas.  Typically,  these  appraisals  are 


This  process  is  explained  on  Defense  Acquisition  University’s  Acquisition  Community  Connection  Web  site  [DoD 
2006b], 
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performed  using  a  SCAMPI  Class  B  method  that  only  addresses  the  processes  proposed  for  the 
program. 

The  acquirer  can  require  the  supplier  to  incorporate  process  appraisals  directed  by  the  acquirer 
into  its  proposed  IMS  and  IMP  and  to  plan  for  cooperation  with  these  appraisals.  The  acquirer 
also  can  provide  instructions  on  approximate  timeframes  for  the  conduct  of  each  appraisal  as  well 
as  information  on  how  far  in  advance  the  supplier  will  be  notified.  Typically,  two  process 
appraisals  are  performed — one  shortly  after  contract  award  and  a  second  later  in  the  program, 
possibly  before  a  major  program  decision  point.  The  first  appraisal  establishes  a  benchmark  of 
process  performance  and  can  be  used  to  encourage  the  supplier  to  expedite  process  instantiation 
on  the  program.  The  second  appraisal  conducted  in  advance  of  a  key  event  can  help  the  acquirer 
assess  the  quality  of  the  efforts  and  products  contributing  to  that  event.  (See  “Consider 
Conducting  Post-Award  Process  Appraisals”  on  page  57  in  Appendix  C  for  implementation 
details.) 

3.14  CONSIDER  PROCESS-BASED,  OUTCOME-ORIENTED  AWARD  FEE 
DETERMINANTS 

A  risk  mitigation  technique,  oriented  to  encourage  the  winning  offeror  to  embrace  process 
implementation,  is  to  negotiate  adding  an  award  fee  determination  based  on  process  adoption  and 
improvement.  If  process  performance  is  included  as  part  of  the  contract  award  fee  conditions  in 
the  RFP,  then  the  acquirer  can  set  the  details  of  the  award  fee  conditions  for  each  offeror  as  part 
of  the  contract  negotiations.  The  acquirer  must  keep  any  award  fee  related  to  process  performance 
in  balance  with  program  outcomes.  According  to  existing  guidance  on  contracts  that  contain 
award  fee  conditions,  “it  is  imperative  that  award  fees  be  tied  to  identifiable  interim  outcomes, 
discrete  events  or  milestones,  as  much  as  possible”  [USD  2006]. 

The  acquirer  may  contact  the  local  DCMA  office  to  request  its  support  with  the  evaluation  of 
award  fee  criteria  implementation  during  contract  execution.  (See  “Consider  Process-Based, 
Outcome-Oriented  Award  Fee  Determinants”  on  page  58  in  Appendix  C  for  implementation 
details.) 

The  methods  presented  in  this  chapter  are  not  mutually  exclusive.  Any  or  all  of  them  may  be  used 
on  a  program.  Table  4  provides  guidance  on  selecting  the  methods  most  appropriate  given 
program  needs. 
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Table  4:  Opportunities  to  Leverage  Supplier  Capabilities — Selection  Guidance 


Title 

Purpose 

Prerequisites 

Outputs 

Impact 

3.1 

Consider  Process 
Proposals  in  Critical 
Areas 

Pages  17  and  40 

Identify  and  understand  the  offeror’s 
intended  application  of  processes  to 
the  program 

Establish  a  basis  for  a  contractual 
obligation  for  process  performance 

Provide  insight  into  artifacts  available 
to  the  acquirer  for  process  monitoring 

Gain  insight  into  organizational 
process  capability 

Process  knowledge  and  experience 
on  the  part  of  the  acquirer  staff  or 
external  support  for  process  proposal 
evaluation 

Proposed  processes 

List  of  available  process  artifacts 

Identification  of  gaps  between  offeror 
capabilities  and  program  needs 

Provides  a  basis  for  contractual 
obligation  for  process  performance 

Requires  significant  effort  from  the 
acquirer  and  may  require  significant 
effort  from  the  offeror 

Use  this  method 

•  on  acquisitions  for  which 
dependence  on  process  capability 
is  significant 

•  for  offerors  whose  process 
capabilities  are  largely  unknown  to 
the  acquirer 

•  to  address  only  the  process  areas 
most  critical  to  the  program 

3.2 

Consider  Integrated 
Master  Plan 
Documentation  of 
Proposed 

Processes 

Pages  18  and  42 

Gain  visibility  of  the  degree  to  which 
offerors  are  planning  to  execute  their 
processes  on  the  program  (i.e., 
contractual  commitment  to  proposed 
processes) 

Defined  offeror  processes  and 
identified  artifacts  that  can  be 
referenced  in  the  IMP  and  IMS 

An  integrated  IMP  and  IMS  that 
incorporate  process  elements  into 
the  program  plan  and  schedule 

Requires  minimal  effort  from  the 
acquirer  and  the  offeror 

3.3 

Consider 

Incorporation  of 
Process  Reviews 
into  Proposed 

Program  Schedules 

Pages  18  and  43 

Provides  the  acquirer  with  visibility 
into  the  deployment  of  processes 
critical  to  the  program 

The  acquirer  has  knowledge  of  the 
expected  execution  of  processes 
critical  to  the  program  within  the 
program's  development  lifecycle 

The  acquirer  has  knowledge  or 
external  support  available  for 
evaluation  of  the  IMS,  IMP,  and 
proposed  process-related  work 
products 

An  IMS  and  IMP  that  include  effort 
and  resources  for  periodic  reviews  of 
processes  critical  to  the  program 

List  of  process-related  work  products 
for  the  processes  critical  to  the 
program 

Requires  minimal  effort  from  the 
acquirer  and  the  offeror 

A  supplier  at  maturity  level  2  or 
higher  is  required  to  conduct  process 
reviews  under  CMMI  requirements. 
While  placing  it  in  the  schedule  may 
increase  the  formality,  costs 
associated  with  this  approach  should 
not  be  significant. 
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Title 

Purpose 

Prerequisites 

Outputs 

Impact 

3.4 

Consider  the 
Approach  to 
Integration  with 
Subcontractor 
Processes 

Pages  19  and  44 

Assure  adequate  integration  and 
coordination  of  prime  contractor  and 
subcontractor  processes 

Subcontract  relationships 

Subcontractor  management  plan 

List  of  subcontractor  integration  risks 

Requires  minimal  effort  from  the 
acquirer  and  the  offeror 

3.5 

Consider  the 
Approach  to 
Integration  with 
Acquirer  Processes 

Pages  19  and  45 

Assure  adequate  integration  and 
coordination  of  the  acquirer  and 
supplier  processes 

Establish  expectations  for 
cooperation  and  integration 

Provide  the  basis  for  creating  a 
contractual  obligation  for  cooperation 
and  integration 

Defined  critical  process  interfaces 

The  supplier  processes  aligned  with 
the  acquirer  processes 

Requires  minimal  effort  from  the 
acquirer  and  the  offeror 

3.6 

Consider  Recent 
Appraisal  Results 

Pages  20  and  46 

Gain  insight  into  the  organizational 
maturity  and  process  capability  of 
offerors 

Appropriate  appraisal  data  available 
(e.g.,  relevant  scope,  organization, 
domain) 

Identification  of  process  assets 
available  to  the  program 

Identification  of  gaps  between  offeror 
capabilities  and  program  needs 

Requires  minimal  effort  from  the 
acquirer  and  the  offeror 

Use  on  acquisitions  where  a 
sufficient  subset  of  potential  offerors 
can  provide  appraisal  data 

3.7 

Consider  Historical 
Data  on  Process 
Performance 

Pages  21  and  51 

Understand  and  identify  risks 
associated  with  the  application  of  the 
offeror’s  process  capabilities  to  new 
programs 

Sufficient  data  on  the  application  of 
mature  processes  to  new  programs 

Process  knowledge  and  experience 
on  the  acquirer  staffs  part  or  external 
support  for  process  proposal 
evaluation 

Insight  into  the  offeror’s  ability  to 
instantiate  organizational  processes 
rapidly  on  new  programs 

Risks  associated  with  the  offeror’s 
consistency  in  applying  mature 
processes  to  new  programs 

Requires  minimal  effort  from  the 
acquirer  and  the  offeror 

Can  provide  an  extremely  useful 
analysis  that  can  differentiate  offerors 
based  on  their  commitment  to  define 
and  execute  capable  processes  on 
new  programs 

Use  if  there  are  doubts  about  the 
intent  of  offerors  to  define  and  apply 
capable  processes  on  the  program 
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Title 

Purpose 

Prerequisites 

Outputs 

Impact 

3.8 

Consider  Recent 
Post-Award 

Appraisal  Data  from 
Other  Programs 

Pages  21  and  53 

Gain  an  improved  understanding  of 
the  supplier’s  process  execution 
capability  on  recent  programs 

Ideally,  the  offeror  had  post  award 
appraisals  performed  on  recent  and 
similar  programs  within  the 
organization  bidding  this  proposal 

Appraisal  data,  including  the 
following 

•  Final  findings  presentation 

•  ADS 

•  Action  plans 

The  collection  and  analysis  of  this 
data  provides  increased  confidence 
in  the  abilities  and  risks  of  the  offeror 
as  demonstrated  by  performance  in 
actual  programs. 

Requires  minimal  effort  from  the 
acquirer  and  the  offeror 

When  this  data  is  applicable  and 
available,  use  this  method  on 
acquisitions  where  processes  have 
been  deemed  to  be  a  critical  factor. 

3.9 

Facilitate  contract  monitoring 
activities  later  in  the  program  by 

None 

Contract  language  establishing  the 
acquirer’s  access  to  relevant  supplier 

Enables  later  contract  monitoring, 
requires  minimal  effort  from  the 

Consider  Process 
Compliance 
Evaluations  During 
Execution 

Pages  22  and  53 

including  provisions  in  the  contract 

Execution  of  the  method  later  in  the 
program  may  require  one  or  more  of 
the  following: 

•  Expertise  in  measurement  and 
analysis  to  monitor  supplier  data 

•  Process  and  appraisal  knowledge 
and  experience  on  the  part  of  the 
staff 

•  External  support  for  measurement 
and  analysis 

•  External  support  for  process  and 
appraisal  activities 

data  and  rights  to  perform  program 
process  appraisals 

acquirer  and  can  greatly  facilitate 
later  monitoring  activities 

Later  execution  of  some  of  these 
activities  (e.g.,  program  process 
appraisals,  data  reviews)  will  impose 
demands  on  the  acquirer;  however, 
judgments  on  proceeding  with  the 
activities  can  be  made  at  that  time 
based  on  program  conditions. 

One  drawback  is  the  slight  increase 
in  cost  to  cover  supplier  support  for 
these  efforts. 

3.10 

Consider  the 
Reporting  of 

Process  Statistics 

Pages  22  and  54 

Ensure  process  compliance  during 
program  execution 

Measurement  and  analysis 
knowledge 

Process  knowledge  and  experience 
on  the  part  of  the  acquirer’s  staff  or 
external  support  for  process  proposal 
evaluation 

Process  audit  reports 

Process  compliance  statistics 

Can  provide  significant  benefit  at  a 
low  cost  to  the  acquirer  and  the 
supplier 

A  supplier  at  maturity  level  2  or 
higher  is  required  to  collect  process 
statistics,  so  additional  costs  should 
not  be  significant. 

3.11 

Consider  Access  to 
Selected  Process 
Artifacts 

Pages  22  and  55 

Ensure  process  compliance  during 
program  execution  through 
monitoring  of  process  artifacts 

Measurement  and  analysis 
knowledge 

Process  knowledge  and  experience 
on  the  part  of  the  acquirer’s  staff  or 
external  support  for  process  proposal 
evaluation 

Process  artifacts 

Indications  of  process  compliance 

Can  provide  significant  benefit  at  a 
low  cost  to  the  acquirer  and  the 
supplier 
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Title 

Purpose 

Prerequisites 

Outputs 

Impact 

3.12 

Consider 

Conducting  Pre- 
Award  Process 
Appraisals 

Pages  23  and  56 

Evaluate  the  offeror’s  organizational 
process  capability,  or  capability  of 
proposed  or  actual  development 
processes 

Differentiate  the  offerors’  process 
capabilities  during  source  selection 

Reinforce  the  commitment  of  the 
supplier  team  to  process  capability 
after  the  contract  award 

Appraisals  must  be  set  through 
source  selection  or  contract  action. 

Process  knowledge  and  experience 
on  the  part  of  the  acquirer’s  staff  or 
external  support  for  process  proposal 
evaluation 

Strengths,  weaknesses,  and  risks  of 
the  offeror’s  organizational 
processes,  proposed  processes,  or 
actual  processes 

A  process  capability  evaluation  of  the 
offeror’s  proposed  process  can 
reveal  lapses  in  intent  or  knowledge 
associated  with  mature  development 
processes.  SCAMPI  Class  C 
appraisals  provide  fairly  inexpensive, 
quick  mechanisms  for  an  in-depth 
understanding  of  the  offeror’s  intent 
that  can  support  comparison  and 
differentiation  of  offerors. 

3.13 

Consider 

Conducting  Post- 
Award  Process 
Appraisals 

Pages  23  and  57 

Ensure  rapid  application  of 
organizational  processes  to  the 
acquirer’s  program 

Ensure  process  compliance  during 
program  execution 

Gain  deeper  insight  into  supplier 
progress 

Process  knowledge  and  experience 
on  the  part  of  the  acquirer’s  staff  or 
external  support  for  process  proposal 
evaluation 

List  of  process-related  risks  to  the 
program 

Supplier  process-compliance  data 

Can  provide  significant  benefit,  but 
the  cost  to  the  acquirer  and  the 
supplier  is  not  trivial 

Use  if  the  supplier  has  a  history  of 
process-related  issues  (e.g.,  slow 
application  of  organizational 
processes,  poor  process 
compliance),  or  if  the  acquirer  has 
other  reasons  to  expect  process 
issues  in  execution 

3.14 

Consider  Process- 
Based,  Outcome- 
Oriented  Award  Fee 
Determinants 

Pages  24  and  58 

Encourage  process  compliance  and 
continuous  improvement 

Process  knowledge  and  experience 
on  the  part  of  the  acquirer’s  staff  or 
external  support  for  process  proposal 
evaluation 

Process-based  outcome-oriented 
award  fee  determinants 

Can  provide  significant  benefit  at  a 
low  cost  to  the  acquirer  and  the 
supplier 
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4  Monitor  Supplier  Process  Performance  After  Award 


This  chapter  provides  guidance  on  how  to  ensure  that  the  supplier  is  effectively  using  capable 
processes  on  the  program,  as  promised  during  solicitation  and  award.  Monitoring  supplier 
execution  after  contract  award  is  a  cornerstone  of  the  acquirer’s  role  as  an  acquisition 
organization. 

Several  methods  can  be  used  to  obtain  the  correct  process  insight  throughout  the  duration  of  the 
contract.  Some  methods  are  inherent  in  a  typical  contract  through  the  submission  and  review  of 
contract  deliverables  that  typically  result  from  process  execution.  Others  involve  deliberate 
review  of  process  documentation  and  artifacts  specifically  involved  in  the  program.  In  addition, 
DCMA  is  a  source  for  performing  on-site  process  monitoring. 

The  methods  used  are  a  function  of  the  program  risks,  the  leverage  provided  within  the  contract 
structure,  and  the  resources  available  for  risk  reduction.  Artifact  reviews  can  be  conducted  on 
formal  contract  deliverables  already  prescribed  for  the  program.  Other  types  of  reviews  involving 
risk  assessments  or  audits  have  been  proven  to  provide  greater  insight  into  process  performance 
and  execution  risks,  but  often  require  enabling  clauses  in  the  contract  to  execute  them. 

Early  invocation  of  these  methods  can  be  a  means  of  jump-starting  a  program’s  process  execution. 
The  methods  are  independent  and  presented  in  no  particular  order — any  or  all  of  them  may  be 
used. 

4.1  PROCESS  MONITORING  THROUGH  ARTIFACT  REVIEWS 

Acquirers  regularly  review  deliverables  (e.g.,  contract  data  requirements  lists  [CDRLs],  status 
reports,  audit  reports)  to  evaluate  perfonnance.  These  same  reviews,  when  linked  to  the  supplier’s 
contractual  processes,  can  also  provide  a  validation  of  both  a  supplier’s  adherence  to  its  processes 
and  its  timely  and  effective  performance. 

Requiring  the  supplier  to  provide  and  commit  to  the  processes  that  it  proposed  and  link  its  work 
products  to  the  IMS,  through  methods  described  in  Chapters  2  and  3,  facilitates  the  ability  to 
reference  its  processes  and  specified  work  products  as  part  of  its  activities  and  to  monitor  the 
completion  of  its  work  products  as  part  of  Earned  Value  Management  (EVM)  reporting.  (See 
“Process  Monitoring  through  Artifact  Reviews”  on  page  6 1  in  Appendix  D  for  implementation 
details.) 

4.2  PROCESS  MONITORING  THROUGH  REVIEW  OF  PROCESS  METRICS  AND 
AUDITS 

Supplier  organizations  should  regularly  gather  metrics  related  to  both  their  program  and  process 
performance  and  perform  periodic  audits  of  process  performance.  A  review  of  these  metrics  and 
audit  results  can  give  the  acquirer  needed  information  on  the  supplier’s  process  implementation 
and  performance.  (See  “Process  Monitoring  through  Review  of  Process  Metrics”  on  page  61  in 
Appendix  D  for  implementation  details.) 
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4.3  PROCESS  MONITORING  THROUGH  PROCESS  REVIEWS 


Early  review  of  the  supplier  team’s  process  capability  reinforces  process  capability  in  low-risk 
teams  and  will  motivate  process  capability  in  high-risk  teams.  Whether  these  reviews  are  initiated 
by  the  acquirer  or  the  supplier,  ideally  they  include  participation  by  both. 

After  contract  award,  the  following  development  process  issues  are  common: 

•  Is  the  supplier  team’s  development  process  defined  and  being  implemented?  If  not,  when 
will  this  happen? 

•  Are  critical  development  processes  in  place  for  program  kick-off? 

•  Has  the  lead  supplier  put  a  plan  in  place  to  integrate  the  team’s  development  processes, 
including  all  subcontractors? 

The  acquirer  can  perform  process  reviews  through  the  access  clause  in  the  contract.  DCMA  can 
perform  them  on  a  continuing  basis  as  part  of  its  contract  monitoring  activities.  (See  “Process 
Monitoring  through  Process  Reviews”  on  page  62  in  Appendix  D  for  implementation  details.) 

4.4  PROCESS  MONITORING  USING  POST-AWARD  APPRAISAL  RESULTS 

Post-award  appraisal  results  (generally,  a  SCAMPI  Class  B  appraisal)  are  characterized  as 
strengths,  weaknesses,  or  gaps,  not  as  capability  levels  or  maturity  levels.  These  findings  provide 
an  accurate  indicator  of  the  state  of  process  areas  that  are  determined  to  be  critical  to  the  program. 

Process  areas  identified  as  low  risk  do  not  require  immediate  action,  but  continued  monitoring  is 
advised.  Those  identified  as  medium  risk  require  further  investigation  to  determine  what  is 
causing  implementation  gaps  and  what  steps  can  be  taken  to  improve.  Those  identified  as  high 
risk  require  immediate  action  as  they  pose  a  significant  barrier  to  successful  deployment  of 
critical  processes.  The  acquirer  can  collaborate  with  the  supplier  on  a  process  improvement  plan 
prioritized  according  to  the  appraisal  findings.  (See  “Process  Monitoring  through  Post-Award 
Appraisals”  on  page  63  in  Appendix  D  for  implementation  details.) 
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Appendix  A:  Questions  and  Checklists 


Table  5:  Acquirer ’s  Decision  Summary 


Selection 

Method  Title 

Responsibility 

Assignment  Date 

Planned 

Implementation  Date 

Actual  Implementation 

Date 

Status 

Identify  the  Critical  Process  Areas  of  the  Program 

□ 

2.3.1  Self  Assessment 

□ 

2.3.2  Facilitated  Assessment 

Leverage  the  Process  Capabilities  of  the  Supplier 

□ 

3.1  Consider  Process  Proposals  in  Critical  Areas 

□ 

3.2  Consider  Integrated  Master  Plan  Documentation  of 

Proposed  Processes 

□ 

3.3  Consider  Incorporation  of  Process  Reviews  into  Proposed 

Program  Schedules 

□ 

3.4  Consider  the  Approach  to  Integration  with  Subcontractor 

Processes 

□ 

3.5  Consider  Approach  to  Integration  with  Acquirer  Processes 

□ 

3.6  Consider  Recent  Appraisal  Results 
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Selection 

Method  Title 

Responsibility 

Assignment  Date 

Planned 

Implementation  Date 

Actual  Implementation 

Date 

Status 

□ 

3.7  Consider  Historical  Data  on  Process  Performance 

□ 

3.8  Consider  Recent  Post-Award  Appraisal  Data  form  Other 

Programs 

□ 

3.9  Consider  Process  Compliance  Evaluations  During 

Execution 

□ 

3.10  Consider  the  Reporting  of  Process  Statistics 

□ 

3.1 1  Consider  Access  to  Selected  Process  Artifacts 

□ 

3.12  Consider  Conducting  Pre-Award  Process  Appraisals 

□ 

3.13  Consider  Conducting  Post-Award  Process  Appraisals 

□ 

3.14  Consider  Process-Based,  Outcome-Oriented  Award  Fee 

Determinants 

Monitor  Supplier  Process  Performance  After  Contract  Award 

□ 

4.1  Process  Monitoring  Through  Artifact  Reviews 

□ 

4.2  Process  Monitoring  Through  Review  of  Process  Metrics 

□ 

4.3  Process  Monitoring  Through  Process  Reviews 

□ 

4.4  Process  Monitoring  Using  Post-Award  Appraisal  Results 
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Appendix  B:  Identify  the  Critical  Process  Areas  of  the 
Program 


The  two  methods  in  this  appendix  can  help  the  acquirer  assess  and  prioritize  the  program’s  critical 
process  areas.  These  methods  may  be  used  individually  or  in  combination.  When  the  acquirer 
chooses  the  method  to  be  deployed  on  the  program,  the  acquirer  staff  assigned  to  execute  this 
decision  may  then  use  this  detailed  implementation  guidance. 

SELF-ASSESSMENT  QUESTIONNAIRE 

The  questionnaire  in  Table  6  is  an  example  of  what  might  be  used  to  identify  the  process  areas 
that  are  important  to  the  program,  using  the  taxonomy  of  CMMI-DEV.  The  questionnaire  is 
amenable  to  execution  on  a  spreadsheet.  By  identifying  those  process  areas  that  are  of  particular 
importance,  the  acquirer  can  focus  on  the  more  critical  processes  and  request  information  related 
to  them.  This  questionnaire  uses  the  specific  goals  of  each  process  area  in  CMMI-DEV  that  were 
discussed  in  Chapter  2. 


Table  6:  Process  Needs  Self-Assessment  Questionnaire 


# 

Process  Area  and  Goals 

Not  Important 

Slightly  Important 

Important 

Very  Important 

Essential 

1 

2 

3 

4 

5 

Project  Planning 

1. 

Establish  Estimates 

O 

O 

O 

O 

O 

2. 

Develop  a  Project  Plan 

O 

O 

o 

O 

O 

3. 

Obtain  Commitment  to  the  Plan 

o 

o 

o 

O 

o 

Project  Monitoring  and  Control 

4. 

Monitor  Project  Against  Plan 

o 

o 

o 

O 

o 

5. 

Manage  Corrective  Action  to  Closure 

o 

o 

o 

O 

o 

Supplier  Agreement  Management 

6. 

Establish  Supplier  Agreements 

o 

o 

o 

O 

o 

7. 

Satisfy  Supplier  Agreements 

o 

o 

o 

O 

o 
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# 

Process  Area  and  Goals 

Not  Important 

Slightly  Important 

Important 

Very  Important 

Essential 

1 

2 

3 

4 

5 

Integrated  Project  Management  +  IPPD 

8. 

Use  the  Project’s  Defined  Process 

O 

O 

O 

O 

O 

9. 

Coordinate  and  Collaborate  with  Relevant 

O 

o 

o 

O 

O 

Stakeholders 

10. 

Apply  IPPD  Principles 

o 

o 

o 

O 

o 

Risk  Management 

11. 

Prepare  for  Risk  Management 

o 

o 

o 

O 

o 

12. 

Identify  and  Analyze  Risks 

o 

o 

o 

O 

o 

13. 

Mitigate  Risks 

o 

o 

o 

O 

o 

Requirements  Management 

14. 

Manage  Requirements 

o 

o 

o 

O 

o 

Requirements  Development 

15. 

Develop  Customer  Requirements 

o 

o 

o 

O 

o 

16. 

Develop  Product  Requirements 

o 

o 

o 

O 

o 

17. 

Analyze  and  Validate  Requirements 

o 

o 

o 

O 

o 

Technical  Solution 

18. 

Select  Product  Component  Solutions 

o 

o 

o 

O 

o 

19. 

Develop  the  Design 

o 

o 

o 

O 

o 

20. 

Implement  the  Product  Design 

o 

o 

o 

O 

o 
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# 

Process  Area  and  Goals 

Not  Important 

Slightly  Important 

Important 

Very  Important 

Essential 

1 

2 

3 

4 

5 

Product  Integration 

21. 

Prepare  for  Product  Integration 

O 

O 

O 

O 

O 

22. 

Ensure  Interface  Compatibility 

O 

o 

o 

O 

o 

23. 

Assemble  Product  Components  and  Deliver  the 

o 

o 

o 

O 

o 

Product 

Verification 

24. 

Prepare  for  Verification 

o 

o 

o 

O 

o 

25. 

Perform  Peer  Reviews 

o 

o 

o 

O 

o 

26. 

Verify  Selected  Work  Products 

o 

o 

o 

O 

o 

Validation 

27. 

Prepare  for  Validation 

o 

o 

o 

O 

o 

28. 

Validate  Product  or  Product  Components 

o 

o 

o 

O 

o 

Configuration  Management 

29. 

Establish  Baselines 

o 

o 

o 

O 

o 

30. 

Track  and  Control  Changes 

o 

o 

o 

O 

o 

31. 

Establish  Integrity 

o 

o 

o 

O 

o 

Process  and  Product  Quality  Assurance 

32. 

Objectively  Evaluate  Processes  and  Work  Products 

o 

o 

o 

O 

o 

33. 

Provide  Objective  Insight 

o 

o 

o 

O 

o 

Measurement  and  Analysis 

34. 

Align  Measurement  and  Analysis  Activities 

o 

o 

o 

O 

o 

35. 

Provide  Measurement  Results 

o 

o 

o 

o 

o 
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What  to  Do 

This  self-assessment  questionnaire  can  be  used  by  the  acquirer  in  the  context  of  the  following 

activities: 

•  Distribute  the  questionnaire  to  key  individuals  within  the  acquirer  staff,  and  ask  them  to 
independently  complete  it. 

•  Convene  a  meeting  of  all  responders. 

•  Discuss  the  evaluations  of  each  item  to  reach  a  consensus. 

•  Record  the  consensus  evaluation. 

FACILITATED  ASSESSMENT 

The  acquirer  can  conduct  a  facilitated  assessment  by  performing  the  following  activities: 

•  Review  program  documentation  to  develop  an  understanding  of  the  program  and  its  risk 
exposure. 

•  Interview  acquirer  staff  to  further  understand  the  program. 

•  Interview  program  stakeholders  to  develop  an  understanding  of  their  needs  and  expectations. 

The  results  of  the  assessment  are  delivered  in  a  report  identifying  and  prioritizing  the  process 

needs  of  the  program,  along  with  the  supporting  rationale. 

What  to  Do 

The  acquirer  arranges  for  a  facilitated  assessment  by  performing  the  following  steps: 

1 .  Contact  the  appropriate  organization  (e.g.,  a  qualified  federally  funded  research  and 
development  center  (FFRDC)  like  the  SEI,  an  SEI  Partner  organization  qualified  to  perform 
appraisals,  or  the  various  SECs  or  SSA  established  within  the  Services)  to  request  assistance 
in  appraising  the  process  needs  of  the  program 

2.  Work  with  the  assigned  assessor10  to  identify  documentation  needs,  knowledgeable  acquirer 
staff,  and  critical  stakeholders. 


10  The  facilitated  assessment  of  critical  process  areas  does  not  have  to  be  performed  using  a  SCAMPI  appraisal 
method;  therefore,  the  term  “assessor”  is  used  here  instead  of  “appraisal  lead.” 
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3.  Provide  the  identified  documentation  to  the  assessor  for  review. 

4.  Work  with  the  assessor  to  schedule  interviews  with  the  acquirer  staff  and  critical 
stakeholders. 

When  to  Do  It 

The  selected  assessment  method 1 1  can  be  used  early  in  the  development  of  the  acquisition 
strategy. 

How  to  Use  the  Results 

Either  method  can  be  used  to  focus  on  the  process  areas  that  are  important,  very  important,  or 
essential  to  the  program  or  to  evaluate  the  process  capabilities  of  potential  offerors.  The  critical 
areas  identified  should  be  part  of  the  RFP  so  that  offerors  can  propose  processes  mapped  to  the 
critical  areas. 


11  The  term  “assessment”  as  used  here  means  that  the  acquirer  hires  someone  from  the  outside  to  determine  the 
critical  process  areas.  No  methodology  or  standard  is  implied. 
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Appendix  C:  Leverage  the  Process  Capabilities  of  the 
Supplier 


This  appendix  provides  implementation  details  for  the  various  approaches  discussed  in  Chapter  3. 
It  covers  various  ways  to  get  a  chosen  approach  into  a  contract  with  a  supplier. 

Traditionally,  all  proposal  information  is  requested  as  part  of  Section  L  of  the  RFP  and  the 
evaluation  of  the  resulting  proposal  submission  is  governed  under  the  provisions  of  Section  M.  In 
cases  where  the  organizations  expected  to  participate  in  the  acquisition  are  known  entities  and  the 
risks  of  process  implementation  are  relatively  well  understood,  this  traditional  approach  works 
well. 

An  alternative  approach,  especially  when  the  potential  offerors  are  not  as  well  known,  is  to 
request  the  process-related  information  as  part  of  a  request  for  information  (RFI)  prior  to  the 
release  of  an  RFP.  This  approach  gains  access  to  the  process-related  information  prior  to  release 
of  the  RFP  so  additional  conditions  might  be  included  in  the  RFP  to  further  reduce  program  risk. 
For  example,  if  an  offeror  is  not  able  to  adequately  demonstrate  that  it  has  adequate  processes  in 
place  and  is  not  involved  in  a  process  improvement  program,  the  risk  to  the  program  may  be  great 
enough  to  require  an  acquisition-related  process  appraisal  as  part  of  the  acquisition  process. 

While  this  alternative  requires  additional  time,  it  may  be  the  only  way  of  gathering  information  on 
process  capability  for  organizations  that  have  not  been  previously  appraised.  It  also  provides  some 
additional  risk  quantification  when  used  to  supplement  the  findings  from  a  previously  performed 
third  party  appraisal.  If  the  acquirer  believes  there  is  enough  process-related  risk  to  the  program, 
an  appraisal  could  be  performed  on  an  offeror  prior  to  the  RFP  release  (e.g.,  as  part  of  an 
Advisory  Multi-Step  Process  prior  to  RFP  release  [Federal  Acquisition  Regulation  Subpart 
15.202]).  If  an  additional  (unanticipated)  offeror  submits  a  proposal,  this  new  offeror  would  be 
subjected  to  the  same  requirements  as  the  other  offerors  (i.e.,  the  appraiser  would  conduct  the 
appraisal  after  receipt  of  the  proposal). 

Guidance  is  also  provided  for  approaches  to  take  when  process-related  risks  associated  with  the 
potential  winner  are  discovered.  Assuming  evaluation  notices  (ENs)  have  been  issued  and 
discussions  have  begun,  the  potential  winner  can  be  requested  to  address  its  weaknesses  in  its  best 
and  final  offer  (BAFO)  and  include  corrections  in  its  final  proposal  document  (FPD).  This  course 
of  action  can  be  considered  when  the  risks  identified  are  severe  enough  to  potentially  increase  the 
award  price.  However,  when  such  action  is  warranted,  particularly  on  programs  in  which  process 
capability  is  deemed  to  be  especially  critical,  having  the  opportunity  to  mitigate  risk  in  advance  of 
award  has  its  benefits. 

Once  the  acquirer  chooses  the  methods  to  be  deployed  on  the  program,  the  acquirer  staff  assigned 
to  execute  these  decisions  may  then  refer  to  the  detailed  implementation  guidance  in  this  appendix 
to  help  them  implement  the  methods. 
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3.1  CONSIDER  PROCESS  PROPOSALS  IN  CRITICAL  AREAS 


The  acquirer  should  be  aware  that  requesting  process  information  can  result  in  large  volumes  of 
information  from  each  offeror.  By  requesting  process  infonnation  the  acquirer  can  rapidly 
accumulate  proposal  page  counts  that  are  daunting  to  review,  and  the  sheer  volume  of  information 
may  overshadow  other  key  aspects  of  the  proposal  that  are  just  as  important  to  evaluate.  The 
acquirer  should  concentrate  on  the  areas  of  critical  concern  by  carefully  selecting  the  information 
needed  and  ensuring  that  it  directly  supports  evaluation  and  decision  making.  Any  process 
information  requested  can  be  tailored  by  the  offeror  for  application  to  the  program.  This  amount 
of  activity  is  charged  to  the  offeror  Bid  and  Proposal  budget,  so  the  acquirer  should  select  the 
process  areas  so  the  amount  of  work  does  not  unnecessarily  drive  up  the  offeror’s  costs. 

Information  on  how  the  offeror  plans  to  tailor  and  deploy  processes  on  the  program  can  be 
provided  in  several  ways  and  to  varying  levels  of  detail.  If  the  acquirer  is  familiar  with  the  offeror 
organization  that  will  be  performing  on  the  program  or  if  a  pre-award  appraisal  has  been 
conducted,  the  appraisal  findings  and  the  offeror  schedule  for  training  and  deployment  of  the 
processes  may  be  sufficient. 

When  the  acquirer  is  less  familiar  with  the  offeror  and  its  process  execution  history,  more  detailed 
information  may  be  necessary.  With  a  focus  on  the  critical  process  areas  for  the  program,  the 
acquirer  should  select  the  types  of  information  required.  This  information  can  be  provided  in  the 
form  of  process  proposals,  which  should  be  the  offeror’s  organizational  processes  tailored 
specifically  for  the  program. 

The  offeror  should  also  provide  process  information  on  major  subcontractors.  Typically,  prime 
contractors  take  two  approaches  with  their  subcontractors: 

1.  have  subcontractors  execute  the  prime  contractor’s  processes  as  team  members 

2.  have  subcontractors  execute  their  own  processes  in  development  of  a  product  to  be  delivered 
to  the  prime 

The  acquirer  should  exercise  caution  in  cases  in  which  prime  contractors  integrate  their  processes 
with  those  of  their  subcontractors.  This  method  may  take  many  months  to  result  in  efficient,  well 
understood,  and  executed  processes  for  the  program.  (See  “Consider  Approach  to  Integration  with 
Subcontractor  Processes”  on  page  44  in  Appendix  C.)  The  offeror  should  be  required  to  describe 
its  approach  and  provide  process,  schedule,  and  task  information  (i.e.,  process  proposals,  IMP, 
IMS,  WBS,  and  plan  for  training  all  team  members)  in  its  proposals.  At  a  minimum,  the  offeror 
should  be  required  to  provide  process  documentation  for  critical  program  process  areas  for  all 
major  subcontractors,  especially  those  that  will  be  executing  their  responsibilities  under  the 
contract  using  their  own  processes.  Depending  on  the  risks  associated  with  the  subcontractor 
portion  of  the  work,  process  proposals  may  also  be  required. 

If  the  offeror  is  using  process  methodologies  other  than  CMMI  and  SCAMPI  (e.g.,  ISO/IEC 
15288,  12207,  15504;  ISO  9001,  EIA  632,  and  IEEE  1220),  or  if  there  are  processes  to  be 
implemented  that  are  not  captured  by  CMMI-DEV  (e.g.,  manufacturing  practices),  the  acquirer 
should  require  the  offeror  to  map  the  information  into  a  format  consistent  with  the  one  being  used 
to  map  processes  to  CMMI-DEV  (as  shown  in  Chapter  2  and  Appendix  B).  This  mapping  should 
include  process  descriptions  and  tailoring  as  well  as  any  formal  audit  results. 
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ISO-9001 :2000  is  a  quality  management  standard  for  development  created  and  maintained  by  the 
International  Organisation  for  Standardisation  (ISO).  The  American  National  Standard  equivalent 
is  ANSI/ISO/ASQ  Q9001-2000.  Organizations  attain  ISO-9001:2000  registration  through  an 
independent  audit  process  that  examines  their  compliance  to  identified  practices.  Because  ISO- 
900 1 :2000  is  primarily  a  quality  standard,  it  is  less  prescriptive  of  the  development  process  than 
CMMI-DEV,  but  it  does  provide  evidence  of  a  process  focus  within  the  organization.  If  the 
bidding  organization  (prime  contractor  or  subcontractors)  claims  ISO  9001:2000  registration,  the 
acquirer  should  determine  if  the  processes  being  bid  are  included.  Since  ISO  9001:2000  is  a 
quality  system  registration,  it  only  includes  those  processes  that  are  specifically  cited.  Therefore,  a 
development  organization  may  have  a  subset  of  its  processes  (e.g.,  assembly  and  test  of  hardware) 
within  its  registration  but  may  not  include  its  full  set  of  development  processes. 

The  information  supplied  in  offeror  process  proposals  can  be  formally  evaluated  and  scored  as  a 
factor  in  source  selection. 

What  to  Do 

The  acquirer  can  request  descriptions  of  the  processes  in  areas  critical  to  the  program  that  must  be 
examined  in  detail  to  thoroughly  assess  the  potential  risk  of  execution.  These  requests  can  be 
made  in  Section  L  of  the  RFP.  The  acquirer  should  include  evaluation  criteria  related  to  process 
performance  and  risk  in  Section  M  of  the  RFP.  Any  process  proposal  information  provided  by  the 
offeror  should  be  above  and  beyond  what  is  included  in  the  software  development  plan  (SDP)  or 
systems  engineering  plan  (SEP),  which  will  have  much  of  the  information  already  provided  for 
evaluation  [DoD  2004].  The  descriptions  can  include  a  list  of  work  products  produced  by  the 
process,  as  well  as  some  samples  of  these  work  products. 

When  to  Do  It 

Collecting  process  proposals  is  the  first  step  in  ensuring  that  the  proposed  processes  will  be 
actually  applied  to  the  program.  If  there  are  concerns  over  the  detailed  implementation  of 
processes  in  specific  areas,  additional  information  can  be  requested.  As  part  of  the  proposal 
evaluation,  the  acquirer  can  then  determine  the  capability  and  suitability  of  these  processes  for  the 
program. 

If  the  acquirer  knows  the  expected  offerors  well,  and  they  have  demonstrated  the  application  of 
capable  processes  to  other  similar  programs,  the  infonnation  can  be  requested  in  the  RFP.  If  the 
acquirer  does  not  know  the  expected  offerors  well,  the  information  can  be  in  an  RFI  for  earlier 
pre-qualification  evaluation. 

How  to  Use  the  Results 

Process  proposals  form  the  basis  for  determining  the  risk  related  to  their  execution  on  the  program 
and  provide  the  basis  for  requiring  adherence  to  those  processes  as  part  of  the  contract.  The 
acquirer  should  request  that  each  prime  contractor  and  major  subcontractor  provide  descriptions 
of  the  processes  that  they  commit  to  use  immediately  after  contract  award. 

One  method  of  evaluating  these  proposed  processes  is  to  use  a  SCAMPI  Class  C  appraisal  prior  to 
award.  This  appraisal  evaluates  the  process  proposals  in  the  critical  program  process  areas  for 
compliance  with  CMMI-DEV.  Because  it  looks  only  at  the  process  proposals  to  determine  the 
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intent  of  the  organization,  this  appraisal  method  is  considerably  less  intrusive,  less  time 
consuming,  and  less  expensive  than  more  comprehensive  appraisals. 

Process  proposals  are  appraised  to  answer  the  following  questions: 

•  Do  the  proposed  processes  reflect  the  best  practices  of  CMMI-DEV? 

•  Are  the  processes,  particularly  those  in  the  process  areas  identified  as  critical  to  program 
success,  complete  and  adequate  to  perform  the  desired  functions? 

•  Are  the  activities  and  work  products  listed  in  the  processes  visible  in  the  proposed  IMS? 

The  result  of  this  appraisal  is  a  collection  of  risks  arising  from  gaps  between  CMMI-DEV  and  the 
proposed  processes.  These  risks  may  be  factored  into  the  source  selection  criteria.  These  process 
proposals  can  also  be  referenced  in  the  contract  of  the  selected  supplier,  thereby  providing  a 
contractual  commitment  to  execute  the  processes  on  the  program,  as  proposed. 

Commitment  to  follow  a  set  of  processes  can  provide  the  acquirer  with  the  means  to  monitor 
supplier  performance  after  award.  Having  process  descriptions,  including  definitions  of  work 
products,  can  also  provide  the  acquirer  with  the  means  to  verify  that  the  supplier  intends  to 
execute  the  processes. 

3.2  CONSIDER  INTEGRATED  MASTER  PLAN  DOCUMENTATION  OF  PROPOSED 
PROCESSES 

What  to  Do 

The  offeror  should  identify  processes  that  map  to  critical  process  areas  identified  for  the  program 
in  the  RFP  and  include  references  to  organizational  process  materials  (or  materials  already 
tailored  for  the  program)  in  the  IMP.  The  offeror  should  also  include  dates  for  development 
program  team  training  (if  any)  and  process  deployment  in  the  IMS.  The  offeror  should  be  required 
to  describe  its  approach  and  provide  process,  schedule,  and  task  information  (i.e.,  process 
proposals,  IMP,  IMS,  WBS,  and  plan  for  the  training  of  team  members)  in  its  proposals.  These 
activities  should  be  traceable  to  the  WBS  as  referenced  in  the  IMP  and  IMS.  A  matrix  can  be 
provided  for  offerors  to  map  their  proposed  processes  to  the  program’s  identified  critical  process 
areas.  In  addition,  offerors  should  map  the  artifacts  of  their  critical  processes  to  the  IMS  and 
identify  the  nomenclature  they  are  using  to  identify  those  artifacts. 

If  the  acquirer  is  familiar  with  the  offeror  organization  that  will  be  performing  on  the  program  or 
if  a  pre-award  appraisal  has  been  conducted,  the  formal  appraisal  findings  and  the  offeror 
schedule  for  training  and  deployment  of  the  processes  may  be  sufficient  to  provide  and 
understanding  of  the  maturity  of  each  offeror’s  set  of  processes. 

When  to  Do  It 

Normally,  the  requirement  to  provide  an  IMP,  IMS,  a  program  WBS,  and  associated  BoE  are 
included  in  the  RFP  along  with  directions  to  complete  the  process  portion  of  the  IMP  and  IMS 
and  any  mapping  matrix. 
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How  to  Use  the  Results 

When  in  receipt  of  an  offeror’s  IMP,  IMS,  WBS,  BoE,  and  mapping  matrix,  the  acquirer  should 
determine  where  the  offeror  identifies  its  program-critical  processes  and  how  it  identifies  the 
artifacts  for  those  critical  processes  in  its  IMS.  The  completeness  of  the  offeror’s  process 
implementation  can  also  be  examined  in  its  process  mapping  against  the  program’s  critical 
process  areas.  The  completeness  of  the  submission  along  with  the  degree  of  support  provided  by 
the  offeror’s  BoEs  can  allow  the  acquirer  to  estimate  the  degree  to  which  each  offeror  is  planning 
to  execute  its  processes  on  the  program. 

3.3  CONSIDER  INCORPORATION  OF  PROCESS  REVIEWS  INTO  PROPOSED 
PROGRAM  SCHEDULES 

What  to  Do 

The  acquirer  should  request  that  the  offeror  include  planned  process  reviews  in  its  IMP  and  IMS 
and  request  that  the  offeror  provide  a  list  of  the  process  work  products  for  the  program’s  critical 
process  areas  that  will  be  included  in  the  following: 

•  EVMS  (if  applicable) 

•  IBR  (if  applicable) 

•  Data  accession  list  (DAL) 

If  the  acquirer  includes  a  request  for  process  reviews  in  the  IMS,  the  acquirer  should  include  this 
requirement  in  the  model  contract  and  in  the  final  contract  so  the  results  can  be  monitored.  If  an 
offeror  has  not  fully  incorporated  process  reviews  into  its  proposed  IMS,  the  acquirer  may  want  to 
direct  them  using  an  EN  to  include  additional  reviews  in  the  supplier’s  final  proposal  submission. 
If  the  acquirer  does  not  include  a  request  for  process  reviews  in  the  IMS,  the  acquirer  can  still 
negotiate  the  inclusion  of  the  reviews  in  the  final  IMS  as  a  risk  reduction  activity  and  introduce 
the  subject  as  part  of  a  review  of  the  IMS  in  discussions. 

When  to  Do  It 

In  Section  L  of  the  RFP,  the  acquirer  should  include  directions  to  incorporate  proposed  process 
reviews  in  the  IMP  and  IMS  and  include  process  work  products  in  the  IMS.  Based  on  the  risks 
related  to  process  determined  earlier,  the  acquirer  should  ask  for  the  appropriate  process  reviews 
to  be  included  immediately  after  contract  award  and  before  the  initial  review  of  the  IMP  and  IMS. 

How  to  Use  the  Results 

The  acquirer  should  answer  the  following  questions  in  reviewing  the  offerors’  proposals: 

1 .  Has  the  offeror  incorporated  appropriate  levels  of  process  reviews  in  its  IMP  and  IMS 
submittals? 

2.  Is  the  first  process  review  soon  enough  after  award  to  allow  the  acquirer  to  determine  if  the 
critical,  early  processes  are  being  employed  effectively? 

3.  Does  the  offeror  propose  appropriate  metrics  that  will  allow  the  acquirer  and  its  management 
to  verify  and  track  the  rollout  of  the  offeror’s  processes? 
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4.  Are  the  key  artifacts  and  work  products  required  by  offeror  processes  identified  and  tracked 
in  the  IMS? 

The  IBR  establishes  the  EVMS  baseline.  A  review  of  process  deployment  could  assess  whether 
the  deployed  processes  are  included  in  support  of  the  baseline,  particularly  through  the  listing  of 
process  work  products  as  milestone  events. 

3.4  CONSIDER  THE  APPROACH  TO  INTEGRATION  WITH  SUBCONTRACTOR 
PROCESSES 

Typically,  prime  contractors  take  two  approaches  with  their  subcontractors: 

1.  Have  the  subcontractors  execute  the  prime  contractor’s  processes  as  team  members. 

2.  Have  the  subcontractors  execute  their  own  processes  in  development  of  a  product  to  be 
delivered  to  the  prime  contractor. 

The  acquirer  should  exercise  caution  in  cases  where  prime  contractors  integrate  their  processes 
with  those  of  their  subcontractors,  as  this  method  may  take  many  months  to  result  in  efficient, 
well-understood  and  executed  processes  for  the  program. 

What  to  Do 

The  acquirer  should  request  each  offeror’s  approach  to  the  integration  of  its  critical  processes  with 
its  subcontractors,  including  the  means  that  the  offeror  proposes  to  eliminate  risks  associated  with 
start  up  and  training.  This  approach  provides  an  indication  of  how  well  this  important  aspect  of 
process  integration  has  been  thought  out.  When  available,  the  offeror  can  provide  an  ADS  or 
appraisal  findings  for  all  subcontractors,  especially  those  executing  their  responsibilities  under  the 
contract  using  their  own  processes.  Depending  on  the  risks  associated  with  the  subcontractor 
portion  of  the  work,  process  proposals  may  also  be  required 

When  to  Do  It 

This  information  is  important  for  any  program  that  includes  a  significant  amount  of 
subcontracting.  If  the  expected  offerors  are  well  known  and  have  demonstrated  the  application  of 
mature  processes  to  other  similar  programs,  this  information  can  be  requested  in  the  RFP.  If  the 
expected  offerors  are  not  well  known,  this  information  can  be  requested  in  an  RFI  for  earlier  pre¬ 
qualification  evaluation. 

How  to  Use  the  Results 

There  are  two  different  situations  that  result  in  two  different  approaches  to  address  the  potential 
risk  of  prime  contractor  and  subcontractor  process  relationships.  Both  can  occur  on  the  same 
acquisition  program  within  different  bidding  teams. 

1 .  If  a  prime  contractor  proposes  that  all  subcontractors,  or  even  some  of  the  critical 

subcontractors  will  use  their  own  processes  to  perform  development,  then  the  evaluation 
team  must  be  prepared  to  determine  whether  the  prime  and  the  subcontractors  have 
substantiated  the  following  about  themselves: 

They  are  sufficiently  mature  in  their  own  right  to  execute  their  processes  effectively. 
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They  have  addressed  in  their  proposal  how  they  will  integrate  their  processes  to 
minimize  duplication  and  effectively  address  all  the  practices  related  to  the  program’s 
critical  processes. 

Unless  the  bidding  team  has  already  analyzed  and  agreed  on  process  interfaces,  a  significant 
risk  exists  that  the  processes  will  not  effectively  execute  from  the  outset.  Considering  that 
early  execution  of  time-critical  processes  most  often  determines  the  success  of  a  program,  it 
is  imperative  that  the  bidding  team  address  those  interfaces  in  detail  prior  to  contract  award 
and  assure  the  evaluation  team  that  it  has  an  effective  approach. 

2.  If  the  prime  contractor  proposes  a  distributed  development  facility  with  each  major 

subcontractor  using  its  own  processes,  the  risk  resides  in  the  ability  of  the  prime  to  deploy  an 
integrated  development  environment  (IDE)  and  to  integrate  its  overarching  processes  with 
the  subcontractors  through  an  organizational  and  technical  interface  structure.  If  the  prime 
intends  to  execute  a  distributed  development  environment,  its  productivity  will  be  lower  at 
the  start  of  the  program  unless  it  plans  to  implement  the  IDE  prior  to  award  and  train  its 
subcontractors  on  the  tools  that  it  intends  to  use. 

If  the  prime  contractor  proposes  a  single  development  site  with  all  subcontractors  using  the 
prime’s  processes,  the  risk  resides  in  the  ability  of  the  prime  to  rapidly  train  and  employ  its 
subcontractors.  If  the  prime  intends  to  execute  this  training  after  the  award,  its  productivity  will  be 
lower  at  the  start  of  the  program  and  early  design  and  architectural  activities  may  incur  additional 
risk  and  effort. 

If  a  prime  contractor  proposes  that  it  will  impose  its  processes  on  subcontractors,  then  it  should 
have  a  plan  for  training  the  subcontractors  on  its  organizational  processes.  If  the  prime  contractor 
proposes  to  do  so  after  award,  then  it  is  imposing  considerable  risk  that  the  early  activities  will  not 
effectively  use  its  processes.  It  is  not  unreasonable  to  expect  a  prime  contractor  to  propose 
instituting  training  of  prospective  subcontractors  prior  to  contract  award.  If  such  training  is  not 
proposed,  then  the  evaluation  team  can  consider  the  early  execution  of  program  processes  to  be  at 
risk,  at  least  at  risk  for  errors  or  omissions. 

In  assessing  the  risk  to  the  program,  the  acquirer  should  determine  whether  the  approaches  chosen 
by  each  offeror  will  add  to  or  mitigate  the  risks  associated  with  effective  program  start  up  or  with 
implementing  effective  processes  critical  to  the  program. 

3.5  CONSIDER  THE  APPROACH  TO  INTEGRATION  WITH  ACQUIRER  PROCESSES 
What  to  Do 

The  acquirer  should  request  each  offeror’s  approach  to  integrating  critical  program  processes  that 
will  be  executed  in  coordination  with  the  acquirer.  This  information  indicates  how  well  each 
offeror  can  address  critical  program  processes  in  concert  with  the  acquirer  processes  after  award. 

When  to  Do  It 

This  information  is  important  for  any  program  that  includes  a  significant  dependence  on  process 
capability.  If  the  expected  offerors  are  well  known  and  have  demonstrated  the  application  of 
mature  processes  to  other  similar  programs,  this  information  can  be  requested  in  the  RFP.  If  the 
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expected  offerors  are  not  well  known,  this  information  can  be  requested  in  an  RFI  for  earlier  pre¬ 
qualification  evaluation. 

How  to  Use  the  Results 

A  detailed  understanding  of  the  acquirer  processes,  and  plans  for  developing  and  improving  those 
processes,  is  a  necessary  first  step.  If  otherwise  appropriate,  it  is  a  good  idea  for  the  acquirer  to 
provide  the  offeror  with  a  brief  description  (or  list)  of  the  processes  it  expects  to  put  in  place  so 
that  the  offerors  can  provide  inputs  or  suggest  additional  processes  that  might  further  reduce  risk 
in  the  overall  program. 

Offerors  can  provide  detailed  benefits  and  approaches  to  acquirer  and  supplier  process  integration. 
The  acquirer  might  consider  the  following  typical  attributes  of  process  integration  as  beneficial: 

•  increased  visibility  into  supplier  operations 

•  enhanced  timeliness  for  the  delivery  or  sharing  of  program  management  information 

•  enhanced  accuracy  and  understanding  of  information 

•  enhanced  ability  of  both  the  acquirer  and  supplier  to  make  (specific  and  appropriate) 
program  decisions 

•  enhanced  plans  for  the  acquirer’s  participation  with  the  supplier  in  integrated  project  teams 
(IPTs)  (Preferably,  such  participation  is  not  limited  to  information  sharing,  but  includes 
shared  decision-making.) 

Typically  the  acquirer  should  expect  a  supplier  to  recommend  the  integration  of  risk  management, 
requirements  management,  requirements  development,  and  configuration  management  processes. 
In  certain  circumstances,  it  might  also  be  appropriate  for  the  acquirer  to  consider  the  integration 
of  project  monitoring  and  control,  measurement  and  analysis,  and  product  integration  (if  the 
acquirer  will  act  as  the  final  or  system  integrator)  processes. 

To  be  considered  viable,  offeror  recommendations  for  process  integration  should  be  defined  well 
enough  to  be  clearly  implemented  in  a  timely  manner.  In  general,  recommendations  for  process 
integration  should  also  address  known  or  identifiable  program  risks. 

3.6  CONSIDER  RECENT  APPRAISAL  RESULTS 

When  available,  data  on  previous  organizational  appraisals  can  be  gathered  from  each  prime 
contractor  and  subcontractor  that  will  perform  major  development  activity. 

With  the  release  of  version  1.2  of  the  SCAMPI  Method  Definition  Document  (MDD),  the  ADS 
must  contain  the  following  information: 

•  the  appraisal  sponsor  and  sponsor’s  organizational  affiliation 

•  the  appraisal  team  leader,  appraisal  team  members,  and  their  organizational  affiliations 

•  the  organizational  unit  appraised 

•  the  projects  and  other  groups,  function  and  placement  within  the  organizational  unit,  and 
their  geographic  locations 
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•  sample  size,  in  quantifiable  terms  of  the  organizational  sample  in  relation  to  the  size  of  the 
organizational  unit 

•  CMMI  model  used  (version,  representation,  and  disciplines) 

•  appraisal  method  used  (name  and  version) 

•  itemization  of  process  areas  rated  and  process  areas  not  rated 

•  maturity  level  and  capability  level  ratings  assigned 

•  goal  ratings  for  process  areas  within  the  scope  of  the  appraisal 

•  dates  of  on-site  activity 

•  date  the  ADS  was  issued  and  period  of  the  validity  of  the  appraisal  results 

•  statement  affirming  that  all  SCAMPI  requirements  were  met 

•  signature  of  appraisal  team  leader  and  sponsor  with  indication  of  agreement  to  publish 
appraisal  results 

The  SCAMPI  ADS  Example  Template,  in  appendix  A  of  the  SCAMPI  MDD,  contains  the 
guidance  provided  to  Lead  Appraisers  on  the  content  of  the  vl.2  ADS  [SEI  2006b].  There  is  a 
requirement  to  define  the  percentage  of  people  and  projects  appraised  in  relation  to  the 
organizational  unit.  There  is  also  a  requirement  to  provide  the  percentage  of  each  critical  factor 
identified  in  appraisal  planning  covered  by  the  organizational  scope  in  relation  to  the 
organizational  unit. 

Examples  of  critical  factors  include  the  following: 

•  application  domains  (i.e.,  lines  of  business) 

•  geographical  breadth 

•  disciplines  (i.e.,  systems  engineering,  software  engineering,  hardware  engineering) 

•  effort  types  (e.g.,  development,  maintenance,  services) 

•  project  types  (e.g.,  legacy,  new  development) 

•  customer  type  (e.g.,  commercial,  DoD,  NASA) 

•  lifecycle  models  in  use  within  the  organization  (e.g.,  spiral,  evolutionary,  waterfall, 
incremental) 

The  acquirer  can  use  this  information  to  determine  the  relative  sample  size  of  the  projects 

appraised  as  part  of  the  ADS  documentation.  For  offerors  that  submit  version  1.1  of  the  ADS,  the 

12 

acquirer  should  ask  for  the  information  in  the  previous  lists. 

The  appraisal  findings  usually  take  the  form  of  a  briefing  containing  an  overview  of  strengths  and 
weaknesses  and  the  final  results  of  the  appraisal  described  in  the  ADS.  Often,  individual  practice 
characterizations  aggregated  to  the  organizational  level  are  also  presented.  At  the  request  of  the 


The  data  requirements  for  the  ADS  have  been  strengthened  with  the  release  of  SCAMPI  Version  1 .2.  An  ADS 
from  a  SCAMPI  appraisal  within  the  last  3  years  is  based  on  the  previous  version  of  the  SCAMPI  method  and 
may  contain  less  information  and  require  further  investigation. 
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appraisal  sponsor,  program-level  data  may  also  be  included  in  the  final  findings.  This  level  of  data 
is  not  found  in  the  ADS. 

What  to  Do 

For  the  prime  contractor  and  for  each  subcontractor  or  team  member  involved  in  major 
development  activity,  request  the  ADS  and  the  appraisal  findings  that  were  generated  as  the  result 
of  any  SCAMPI  Class  A  appraisals  within  the  last  3  years.  This  documentation  can  be  requested 
for  subcontractors  as  well  as  for  the  prime.  The  most  reliable  data  are  from  the  most  recent 
appraisal.  Any  supplier  claiming  a  maturity  level  in  its  proposal  should  be  willing  to  release  this 
documentation. 

The  acquirer  can  also  request  the  offeror  to  specify  the  exact  relationship  of  the  organization  that 
was  appraised  to  the  bidding  organization.  Was  the  bidding  organization,  and  specifically  projects 
within  the  bidding  organization,  within  the  scope  of  that  appraisal?  If  not,  will  the  bidding 
organization  be  using  the  processes  that  were  within  the  scope  of  the  appraisal? 

Determining  whether  or  not  the  organization’s  ADS  represents  the  organization  as  a  whole  and 
whether  or  not  it  represents  the  team  executing  the  program  is  important.  Methods  used  to  sample 
the  organization  are  varied  and  have  in  the  past  not  been  governed  by  any  strict  rules  or  disclosure 
requirements. 

The  acquirer  should  request  information  on  the  number  of  active  development  projects  at  the  site 
being  proposed  to  perform  the  work,  the  number  of  active  development  projects  contained  in  the 
bidding  organization  (if  different),  and  the  number  of  active  development  projects  contained  in 
the  organization  that  was  appraised  (if  different). 

Each  offeror  (prime  contractor  and  major  subcontractor)  that  provides  ADS  data  can  also  be 
required  to  provide  information  on  the  number  of  major  development  projects  that  are  currently 
under  contract  in  the  appraised  organization.  Additionally,  the  acquirer  should  request  the  number 
of  staff  members  assigned  and  the  revenue  for  each  project  (e.g.,  major  development  programs 
may  be  defined  as  those  greater  than  18  months  in  duration  and  greater  than  $10  million  in  total 
contract  size). 

When  reviewing  the  ADS,  the  acquirer  should  determine  the  number  and  identity  of  the  programs 
appraised  in  the  ADS  and  compare  them  to  the  active  major  development  program  totals 
provided.  This  determination  should  provide  a  good  estimate  of  how  reasonable  the  sampling  was 
and  how  representative  the  appraisal  is  of  the  organization.  It  will  also  provide  some  idea  of  the 
sampling  rates  that  the  bidding  organizations  used  for  their  appraisals.  For  more  recent  appraisals, 
this  information  will  be  included  within  the  ADS.  Regardless,  it  is  prudent  for  the  acquirer  to 
pursue  this  issue  to  ensure  that  sampling  used  by  the  appraised  organization  is  understood. 

When  to  Do  It 

By  requesting  this  information  in  an  RFI  preceding  the  formal  RFP  release,  the  acquirer  can 
examine  the  potential  offerors’  organizational  maturity  and  process  capability.  This  information 
can  then  help  fine-tune  the  evaluation  criteria.  Issuing  such  an  RFI  also  puts  potential  offerors  on 
notice  that  process  capability  is  a  concern.  Such  notice  may  discourage  non-process-focused 


48  |  CMU/SEI-2007-TR-004 


offerors  from  participating  in  the  RFP,  or  may  encourage  them  to  take  significant  action  to  rectify 
their  shortfalls. 

Offerors  can  be  instructed  (in  Section  L  of  the  RFP)  to  provide  this  information  as  part  of  their 
proposal  submissions.  Evaluation  criteria  can  be  included  in  Section  M  of  the  RFP.  Phrase  these 
evaluation  criteria  in  the  form  of  risk  assessments — risks  posed  to  the  program  as  a  result  of  the 
process  proficiency  of  offeror  organizations  especially  in  the  process  areas  of  key  importance  to 
the  program.  Special  consideration  can  be  given  to  the  process  areas  that  must  be  enabled  from 
the  start  of  the  program,  as  performance  in  the  early  stages  of  the  program  is  critical  to  overall 
success. 

How  to  Use  the  Results 

The  acquirer  should  use  the  data  supplied  in  the  ADS  to  address  the  following  questions: 

1 .  Is  the  organization  named  in  the  ADS  the  same  organization  that  is  bidding  on  and  will 
execute  the  program? 

If  not,  ask 

a.  Is  the  organization  named  in  the  ADS  at  a  higher  level  in  the  organizational  structure 
than  the  organization  bidding  on  and  planning  to  execute  the  program? 

•  If  not,  investigate  the  relationship  between  the  two  organizations.  If  the  relationship 
is  tenuous,  the  information  in  the  ADS  is  probably  not  useful. 

•  If  so,  then  investigate  the  current  relationship  between  the  two  organizations.  Has  the 
organization  been  reorganized  since  the  appraisal?  If  so,  there  could  be  some 
differences  in  the  organizational  commitment  to  the  processes  to  be  employed  that 
must  be  verified 

2.  Was  the  bidding  organization  specifically  involved  in  the  appraisal? 

a.  Was  the  site  proposed  to  perfonn  the  work  represented  by  a  program  in  the  appraisal? 

3.  Was  the  appraisal  perfonned  by  a  third  party,  not  affiliated  with  the  organization  or  the 
company,  or  by  a  Lead  Appraiser  associated  with  the  appraised  organization? 

Many  organizations  employ  SEI-authorized  Lead  Appraisers  as  well  as  other  staff  members 
trained  in  CMMI  appraisal  methods.  Many  organizations  also  contract  with  process 
improvement  consultants  who  are  also  SEI-authorized  Lead  Appraisers,  to  guide  them  on 
their  quest  for  high  process  maturity.  Publicly  reported  appraisals  that  are  led  by  independent 
appraisal  leads  and  staffed  largely  by  independent  appraisers  are  generally  considered  to 
reflect  a  high  confidence  level  for  findings.  This  does  not  mean  that  the  team  cannot  contain 
some  members  of  the  organization.  Internal  team  members  are  considered  an  especially 
valuable  resource  for  interpreting  organizational  structures  and  obtaining  artifacts  during  an 
appraisal;  however,  a  higher  percentage  of  independent  members  on  the  appraisal  team  is 
generally  considered  to  reflect  a  higher  confidence  level  for  findings. 

4.  How  many  projects  were  included  in  the  appraisal?  (Use  some  of  the  supplemental 
information  to  support  this  analysis.) 

a.  What  was  the  size  of  the  organization  stated  in  the  scope? 

b.  What  criteria  were  used  to  select  programs  to  be  included  in  the  appraisal? 
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c.  How  many  development  programs  are  currently  active  in  that  organization? 

d.  What  is  the  size  of  the  bidding  organization  where  the  work  will  be  performed? 

5.  Was  the  appraisal  performed  less  than  two  years  ago? 

It  is  unlikely  that  an  organization  that  is  actively  involved  in  process  improvement  would 
leave  all  its  processes  untouched  for  more  than  two  years.  In  organizations  at  maturity  or 
capability  level  3  and  higher  (and  occasionally  at  lower  levels),  process  assets  should  be 
gathered  in  an  organizational  repository  and  be  placed  under  change  control.  Change  records 
from  this  repository  are  a  clear  indicator  of  process  change.  Review  of  these  records  provides 
two  pieces  of  information: 

a.  An  indication  of  the  activity  in  the  process  repository.  A  noticeable  absence  of 
submissions  or  revisions  may  be  an  indication  of  a  lack  of  process  focus 

b.  A  view  into  the  changes  in  the  processes  since  the  last  appraisal.  Substantial  changes  in 
the  processes  may  invalidate  the  findings  of  prior  appraisals.  Explanations  and 
motivations  for  the  changes,  as  well  as  discussions  of  the  applicability  of  the  prior 
appraisals  can  be  requested. 

6.  What  was  the  appraisal  method  used? 

a.  If  the  offeror  is  claiming  maturity  level  or  capability  level  ratings,  expect  that  the 
appraisal  data  submitted  is  from  a  formal  SCAMPI  Class  A  appraisal.  Organizations 
have  been  known  to  wrongly  claim  ratings  based  on  SCAMPI  Class  B  appraisals  or 
“internal  assessments  of  their  process.”  Neither  employs  the  rigor  of  a  SCAMPI  Class 
A  appraisal,  particularly  related  to  artifacts  and  interviews,  and  is  not  approved  for 
bestowing  a  rating. 

b.  Two  SCAMPI  method  versions  and  models  might  have  been  employed.  Prior  to  August 
2006  and  for  a  sunset  period  until  September  2007,  version  1.1  can  be  used.  After 
August  2006,  the  latest  version  of  the  model,  CMMI-DEV,  V 1 .2,  and  the  corresponding 
method,  is  preferred.  After  November  1,  2006,  v  1.2  of  the  ADS  must  be  used 

7.  Which  process  areas  were  included  in  the  appraisal?  Organizations  can  choose  to  appraise 
using  one  of  two  model  representations,  staged  or  continuous.  Staged  appraisals  require 
certain  sets  of  process  areas  to  be  appraised  to  achieve  a  maturity  level  rating  and  success  is 
designated  by  an  indication  that  the  goals  of  the  process  area  are  “satisfied”.  Continuous 
appraisals  require  that  each  process  area  included  in  the  appraisal  be  examined  and  a 
capability  level  be  designated  for  each  process  area.  When  viewing  the  results  of  an 
appraisal,  it  is  important  to  determine  if  the  specific  process  areas  that  were  designated  as 
critical  to  the  program  were  included  in  the  appraisal. 

a.  Prior  to  vl  .2,  organizations  using  the  staged  representation  were  allowed  to  designate 
some  process  areas  as  NA.  In  practice,  the  NA  designation  is  most  often  applied  to  the 
Supplier  Agreement  Management  (SAM)  process  area.  In  vl.2  and  later,  only  Supplier 
Agreement  Management  (SAM)  can  be  declared  NA,  and  only  for  organizations  that  do 
not  have  significant  supplier  inputs.  If  SAM  was  not  included  in  the  appraisal,  the 
prime’s  process  for  dealing  with  critical  subcontractors  has  not  been  appraised  and  the 
risk  associated  with  managing  subcontractor  activities  should  be  examined  in  more 
detail. 
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8.  What  is  the  organizational  significance  of  the  appraisal  sponsor?  Is  the  sponsor’s  position 
appropriate  for  the  declaration  of  the  appraisal  scope?  If  multiple  organizations  are 
represented  in  the  appraisal,  is  the  sponsor  in  the  overarching  organization  and  able  to  attest 
to  the  commitment  and  policy  of  the  overarching  organization? 

Use  the  appraisal  findings  to  answer  the  following  questions: 

1 .  Do  appraisal  findings  match  the  scope  of  the  ADS,  or  do  they  document  a  modification  of 
the  appraisal  scope?  The  ADS  might  report  that  a  program  initially  included  was  not  fully 
involved  or  provided  only  part  of  the  information.  Programs  are  only  expected  to  provide 
artifacts  that  demonstrate  the  effort  expended  at  the  stages  in  the  lifecycle  that  they  have 
actually  executed.  For  example,  newer  programs  may  not  have  test  and  evaluation  data  or 
complete  product  integration  data.  In  such  cases,  the  appraisal  team  may  use  the  term  not  yet 
which  contains  no  negative  implications  to  the  findings. 

2.  Do  the  appraisal  findings  declare  any  process  areas  NA?  If  any  of  the  process  areas  declared 
NA  are  related  to  critical  program  processes,  then  there  would  be  additional  risk  to  the 
program  in  selecting  an  organization  that  has  not  demonstrated  capability  in  a  critical  area. 

3.  Will  weaknesses  noted  in  the  appraisal  findings  have  a  significant  impact  on  the  program? 
Has  the  organization  addressed  or  corrected  any  of  these  noted  weaknesses?  Has  the 
sufficiency  of  the  corrections  been  evaluated?  Appraisal  findings  are  required  to  list  any 
critical  weaknesses  found.  Critical  weaknesses  are  those  that  keep  a  maturity  or  capability 
level  from  being  achieved  and  will  be  clearly  indicated  in  the  findings.  Identified  strengths 
and  weaknesses,  especially  critical  weaknesses,  can  be  evaluated  against  the  program’s 
required  process  areas,  particularly  when  they  are  identified  in  the  portion  of  the 
organization  that  would  be  performing  on  the  program. 

Use  the  publicly  reported  appraisal  data  to  corroborate  the  ADS  information  on  the  PARS  [SEI  2]. 
If  discrepancies  are  found  between  the  publicly  reported  data  and  the  data  supplied  in  the  proposal 
response,  additional  requests  for  information  may  be  required. 

3.7  CONSIDER  HISTORICAL  DATA  ON  PROCESS  PERFORMANCE 

Reviewing  the  results  of  past  appraisals  is  only  one  way  of  examining  the  history  of  process 
application  within  an  organization.  The  appraisal  process  provides  an  in-depth  examination  of  the 
process  utilization  on  a  selected  sample  of  projects  either  completed  or  in-process.  However,  due 
to  the  intrusive  nature  of  the  appraisal  process,  it  is  not  practical  to  examine  more  than  a  few 
programs  within  an  organization.  While  the  chosen  programs  are  expected  to  be  representative  of 
the  broader  organization,  they  may  constitute  an  extremely  small  sample,  and  it  may  be 
impossible  to  verify  their  application  throughout  the  remainder  of  the  organization. 

An  alternative  method  of  examining  the  history  of  process  application  within  an  organization  is  to 
review  that  organization’s  own  records  of  process  application.  Many  higher  maturity 
organizations  collect  data  on  the  application  and  performance  of  their  processes. 

This  data  typically  includes  metrics  that  track  the  following: 

•  the  organizational  processes  applied  to  a  program 
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•  modifications  to  the  organizational  processes  required  for  a  program 

•  compliance  of  program  activities  with  the  applied  processes 

If  this  data  is  available,  it  can  be  valuable  to  provide  insight  into  the  process  focus  of  the 
organization,  the  consistency  of  process  application  within  the  organization,  and  the  monitoring 
and  enforcement  of  processes  within  the  organization. 

Most  solicitations  request  examples  of  performance  on  past  contracts.  Looking  for  correlations 
between  the  examples  provided  in  the  supplier’s  proposal  and  projects  participating  in  prior 
process  appraisals  can  also  be  useful. 

What  to  Do 

Request  information  detailing  the  historical  use  of  organizational  processes  on  past  and  current 
projects.  In  particular,  the  acquirer  should  seek  statistics  on  the  following: 

•  the  breadth  of  process  application  across  the  organization  (e.g.,  what  percentage  of  projects 
employ  standard  or  tailored  versions  of  the  organization’s  standard  processes) 

•  the  consistency  of  process  application  (e.g.,  what  percentage  of  processes  applied  to  projects 
are  modified  from  the  organization’s  standard  processes) 

•  project  compliance  with  the  defined  processes  (e.g.,  metrics  resulting  from  process 
monitoring  and  process  audits) 

When  to  Do  It 

The  optimal  time  to  request  most  of  this  information  is  before  the  formal  RFP  is  distributed.  If  the 
acquirer  has  a  detailed  understanding  of  the  process  capabilities  of  potential  offerors,  it  can 
understand  the  potential  risk  of  selecting  an  offeror  with  immature  development  processes.  An 
analysis  of  the  information  provided  allows  the  acquirer  to  structure  the  risk  evaluation  during 
technical  evaluation  and  source  selection. 

If  appropriate  to  the  evaluation,  the  acquirer  can  request  that  the  offerors  indicate  which  programs 
provided  for  past  performance  evaluations  also  have  historical  process  data. 

How  to  Use  the  Results 

Historical  data  of  process  performance  on  programs  in  the  same  (or  similar)  domain  as  the 
program  being  bid  is  more  relevant  and  more  likely  to  be  a  better  predictor  of  potential 
performance. 

The  most  desirable  situation  regarding  the  consistency  of  process  application  (percentage  of 
processes  actually  applied  to  projects)  is  that  all  or  most  programs  in  the  organization  actually 
apply  tailored  versions  of  the  standard  processes.  Historical  data  reflecting  process  capability  for 
programs  at  multiple  points  in  their  lifecycle  is  a  clear  indicator  of  an  organization’s  commitment 
to  the  implementation  of  standard  processes  in  the  development  of  their  work  products. 
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3.8  CONSIDER  RECENT  POST-AWARD  APPRAISAL  DATA  FROM  OTHER 
PROGRAMS 

What  to  Do 

The  acquirer  should  request  all  available  post-award  appraisal  information  on  any  recent  (i.e., 
within  two  years)  government-mandated,  post-award  appraisals  of  the  bidding  organization  and 
sites  proposed  for  development  activities. 

When  to  Do  It 

The  optimal  time  to  request  most  of  this  information  is  before  the  formal  RFP  is  distributed,  but 
this  information  may  be  helpful  at  any  time  during  the  life  of  the  program. 

How  to  Use  the  Results 

During  the  last  year  or  so,  acquiring  organizations  have  often  requested  post-award  appraisals  of 
winning  organizations.  If  the  prime  contractor  or  key  development  subcontractors  have  had  a 
post-award  appraisal  on  a  newly  awarded  program,  the  acquirer  should  use  the  results  of  these 
appraisals  to  answer  the  following  questions: 

1 .  Has  the  post-award  appraisal  result  shown  that  the  organization  adopts  its  processes  rapidly 
and  effectively? 

2.  Has  there  been  more  than  one  instance  of  post-award  appraisals  on  a  single  project?  Did  the 
sequence  of  appraisals  demonstrate  progress  in  resolving  process  deficiencies? 

3.  Were  the  results  indicative  of  effective  deployment  or  contradictory  to  organizational 
commitment  to  effective  process  deployment? 

3.9  CONSIDER  PROCESS  COMPLIANCE  EVALUATIONS  DURING  EXECUTION 

Some  organizations  have  process  compliance  evaluations  as  part  of  regular  product  and  process 
quality  assurance  reviews  and  others  make  those  evaluations  part  of  a  more  global  Mission 
Assurance  review  process.  Regardless  of  how  the  individual  organization  performs  that  function, 
it  benefits  the  acquirer  to  have  access  to  the  process  reviews  performed  on  the  specific  program 
for  which  it  is  responsible.  Most  process-focused  organizations  welcome  customer  involvement  in 
their  process  program,  primarily  because  it  improves  understanding  of  the  process  issues  that 
would  improve  program  performance. 

What  to  Do 

In  preparing  the  RFP,  the  acquirer  should  include  a  requirement  in  “Instructions  for  Preparation  of 
the  Offeror’s  Proposal”  in  Section  L  to  confirm  agreement  to  the  acquirer’s  participation  in 
process  compliance  evaluation  actions  and  request  the  offeror  to  propose  how  that  participation 
would  best  fit  its  organizational  process  enforcement  approach. 

Typically,  DCMA  relies  on  the  FAR  standard  inspection  (access)  clause  to  gain  access  to  the 
contract,  supplier,  products,  services,  data,  etc.  This  approach  is  sufficient  for  many  contract 
monitoring  purposes.  If  the  acquirer  intends  to  have  a  support  party,  such  as  DCMA  or  an 
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FFRDC,  be  involved  in  verifying  process  compliance,  that  intention  can  be  identified  in  the  RFP 
so  agreements  can  be  made  regarding  access  to  information  by  the  support  party  prior  to  award. 

When  to  Do  It 

The  acquirer  can  lay  the  groundwork  for  participation  in  the  process  compliance  evaluations  as 
part  of  the  RFP.  By  making  the  request  when  competition  is  paramount,  the  offerors  should  be 
more  willing  to  meet  the  request.  An  additional  option  is  to  evaluate  the  offeror’s  response  to  the 
request,  include  it  in  any  ENs  created,  and  resolve  any  issues  as  part  of  the  discussion  process. 

How  to  Use  the  Results 

During  the  proposal  evaluation,  if  the  acquirer  is  proactive,  it  can  use  the  willingness  of  an  offeror 
to  ensure  that  processes  are  executed  on  the  new  program  and  to  provide  visibility  during 
execution  as  an  indication  of  lower  risk  and  a  higher  probability  of  success. 

3.10  CONSIDER  THE  REPORTING  OF  PROCESS  STATISTICS 
What  to  Do 

The  acquirer  may  simply  request  that  the  supplier  furnish  the  process  metrics  that  are  collected  for 
the  program.  If  the  supplier’s  response  is  inadequate,  additional  metrics  can  be  requested.  For 
example,  some  or  all  of  the  common  process  statistics  listed  below  may  be  applicable  to  the 
program;  however,  the  acquirer  should  consider  the  supplier’s  effort  needed  for  reporting  of  these 
metrics: 

•  process  implementation  timelines  (i.e.,  how  long  from  project  kick-off  did  it  take  to 
implement  and  begin  using  a  process) 

•  number  (or  percent)  of  the  organization’s  standard  processes  being  adopted  (tailored  for  use) 
by  the  project 

•  number  (or  percent)  of  work  products  required  by  a  process  that  were  actually  produced 

•  number  (or  percent)  of  supplier  processes  actually  reviewed  during  or  after  execution  for  the 
purpose  of  identifying  problems,  providing  lessons  learned,  or  for  improving  the  process 

•  number  of  process  defects  identified  over  the  project  lifecycle 

•  percent  of  process  defects  that  were  corrected  (closed) 

•  number  of  project  processes  found  to  be  compliant  with  CMMI-DEV  (at  each  capability 
level) 

•  number  of  process  audits  performed  on  the  developer’s  processes,  for  each  process 

•  number  of  improvements  made  to  the  developer’s  processes,  for  each  process 

When  to  Do  It 

The  reporting  of  process  metrics  or  statistics  can  be  requested  in  the  RFP,  or,  if  omitted,  may  be 
easily  requested  before  contract  award. 
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How  to  Use  the  Results 

A  mature,  well-defined,  and  understandable  set  of  process  metrics  and  statistics  is  desirable.  The 
timeliness  of  process  metrics  can  support  continuous  insight  into  the  supplier’s  process  capability 
and  may  also  provide  insight  into  its  process  readiness  to  support  major  program  milestones. 

The  technical  means  and  timelines  associated  with  providing  process  metrics  to  the  acquirer 
should  be  well  defined  and  realistic.  A  combination  of  independent  process  metrics  into  key 
process  indicators  is  helpful,  as  are  process  metrics  that  measure  the  adequacy  or  performance  of 
integration  with  the  acquirer  processes,  where  appropriate.  Process  control  metrics  should  be 
found  in  high  maturity  organizations  (i.e.,  those  with  maturity  or  capability  level  ratings  of  4  or 
5).  Failure  of  the  supplier  to  provide  meaningful  process  metrics  and  statistics  increases  program 
risk.  Ideally,  the  supplier  has  a  process  for  developing  metrics  and  then  using  them  for  specific 
periods  or  when  triggered  by  certain  events  instead  of  simply  applying  the  same  set  of  metrics  to 
each  program. 

3.1 1  CONSIDER  ACCESS  TO  SELECTED  PROCESS  ARTIFACTS 

Process  performance  can  be  monitored  through  the  evaluation  of  the  artifacts  produced  by  those 
processes  when  applied  to  the  program.  The  artifacts  needed  for  process  performance  evaluation 
are  accessible  to  the  acquirer  by  doing  one  or  more  of  the  following: 

•  Reference  them  in  the  IMS  and/or  IMP 

•  Reference  them  on  the  DAL 

•  Reference  them  on  the  CDRL 

•  Participate  in  integrated  process  teams  with  the  supplier 

•  Attend  peer  reviews  or  other  program  reviews 

Process  implementation  indicators  (PIIs),  which  suppliers  may  collect  for  their  own  SCAMPI 
appraisals,  can  be  used  if  they  are  from  the  actual  program. 

What  to  Do 

The  acquirer  should  request  that  the  supplier  illustrate  process  performance  by  identifying 
artifacts  produced  through  the  execution  of  the  program  processes,  including  the  process 
descriptions  for  the  critical  process  areas  of  the  program.  The  acquirer  then  should  review  the 
process  documents  to  identify  work  products  that  may  be  useful  in  assessing  process  performance. 
Periodically,  the  acquirer  should  obtain  copies  of  the  identified  work  products  from  the  supplier 
and  analyze  them  to  ascertain  the  degree  of  deployment  of  processes  on  the  program. 

When  to  Do  It 

The  RFP  can  establish  the  requirements  for  the  offerors  to  share  their  process  descriptions  and 
provide  access  to  relevant  work  products.  Access  to  the  work  products  can  be  addressed  through 
the  CDRL  and  the  DAL. 

Process  artifacts  can  be  monitored  throughout  the  duration  of  the  acquisition.  In  the  course  of  the 
acquisition,  different  process  areas  vary  in  their  level  of  importance.  When  reviewing  artifacts  for 
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process  performance,  the  acquirer  should  choose  the  processes  that  are  most  significant  at  the 
current  time  given  the  phase  of  the  program. 

How  to  Use  the  Results 

If  the  supplier  is  contractually  obligated  to  perform  specific  processes,  the  acquirer  should  first 
assess  the  impact  of  any  non-compliance  and  address  the  non-compliance  issue  with  the  supplier 
to  determine  any  justification  for  the  discrepancy.  If  reasonable  justification  does  not  exist  and  the 
impact  on  the  program  is  significant,  the  acquirer  should  first  try  to  address  the  issue  informally 
with  the  supplier,  ensuring  the  understanding  of  both  the  requirements  of  the  contract  and  the 
acquirer’s  expectations.  If  this  does  not  resolve  the  issue,  the  acquirer  should  consider  directing 
the  supplier  to  comply  with  stated  contractual  requirements  using  formal  contracting  actions. 

3.12  CONSIDER  CONDUCTING  PRE-AWARD  PROCESS  APPRAISALS 
What  to  Do 

The  process  appraisals  referenced  in  this  guidebook  are  adapted  (tailored)  to  the  situations  most 
useful  to  acquisition,  generally  limited  to  SCAMPI  Class  B  and  Class  C  appraisals.  A  SCAMPI 
Class  B  appraisal  team  may  be  as  small  as  two  or  the  investigation  team  may  be  expanded  to 
include  other  stakeholders  such  as  representatives  of  the  user  community,  supplier  process 
engineers,  or  other  affected  organizations.  The  design  of  the  appraisal  allows  for  a  focused 
productive  dialog  structured  by  the  relevant  process  areas  in  CMMI-DEV.  A  SCAMPI  Class  C 
appraisal  may  be  conducted  by  one  or  more  qualified  and  trained  individuals.  The  output  of  either 
of  these  appraisal  methods  is  a  list  of  findings  from  the  appraisal  and  a  list  of  risks  that  these 
findings  pose  for  the  project. 

In  general,  if  the  acquirer  desires  to  perfonn  an  appraisal,  it  does  not  need  appraisal  expertise;  the 
acquirer  can  work  with  a  trained  and  qualified  appraisal  lead  to  conduct  the  type  of  appraisal 
desired.  The  appraisal  lead  works  with  the  sponsor  (normally  a  senior  manager  of  the  organization 
to  be  appraised)  to  clarify  the  needs  and  requirements  for  the  appraisal.  The  appraisal  lead  may 
then  recommend  various  approaches  to  fulfill  the  requirements.  Once  an  approach  is  chosen,  the 
appraisal  lead  begins  to  develop  detailed  plans  and  estimates  for  approval  by  the  sponsor. 

To  employ  these  appraisals,  the  acquirer  should  use  the  following  key  steps: 

1 .  Determine  that  an  appraisal  is  warranted. 

2.  Work  with  a  qualified  appraisal  lead  to  do  the  following: 

a.  Define  the  goals  and  objectives  of  the  appraisal.  An  experienced  appraisal  lead  can  be 
helpful  in  ensuring  that  the  design  and  execution  of  the  appraisal  delivers  the  expected 
information  and  value. 

b.  Develop  a  detailed  appraisal  plan.  The  appraisal  lead  works  with  the  sponsor  to 
translate  the  goals  of  the  appraisal  into  a  detailed  appraisal  plan  with  estimates  of  time 
and  resources. 

c.  Determine  the  size  of  the  appraisal  team;  appraisals  typically  require  an  appraisal  team 
(i.e.,  individuals  in  addition  to  the  appraisal  lead).  The  team  size  and  qualifications  of 
the  team  members  have  an  impact  on  the  time,  cost,  and  depth  of  the  investigation. 
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Team  members  can  include  acquirer,  user,  supplier/offeror,  or  external  personnel.  The 
appraisal  lead  works  with  the  sponsor  and  project  stakeholders  to  develop  a  staffing 
strategy  appropriate  to  the  type  of  appraisal  and  the  goals  of  the  sponsor. 

d.  Be  clear  about  the  outputs  of  the  appraisal  and  how  they  will  be  used  to  satisfy  the  goals 
of  the  sponsor. 

3.  Approve  the  appraisal  plan. 

4.  Arrange  for  appraisal  resources. 

5.  Help  coordinate  with  appraised  entities. 

6.  Analyze  the  data. 

The  appraisal  team  collects  data  from  the  organization.  This  data  can  be  documents,  work 
products,  processes,  written  affirmations,  and  interviews.  The  appraisal  team  uses  CMMI-DEV  as 
a  framework  for  collecting  and  evaluating  the  evidence  provided  by  the  organization. 

Typical  outputs  from  the  appraisal  include  the  following: 

1 .  Statements  of  strengths  and  weaknesses  mapped  to  model  practices 

2.  Practice  characterizations,  typically  of  risk  of  level  of  compliance 

3.  Other  information  requested  by  the  sponsor  or  deemed  important  by  the  team 

When  to  Do  It 

These  on-site  appraisals  must  be  scheduled  to  support  source  selection  needs  within  the  planned 
selection  period.  Planning  for  the  use  of  these  methods  often  precedes  the  release  of  the  RFP,  and 
depends  on  how  the  results  of  the  appraisals  are  fed  into  the  source  selection  process.  If  results  of 
the  appraisals  are  an  independent  risk  factor,  for  example,  timing  of  the  appraisals  may  be 
flexible.  But  if  the  results  of  the  appraisals  feed  other  teams,  the  appraisals  may  need  to  be 
scheduled  relatively  early  in  the  source  selection  period. 

How  to  Use  the  Results 

Frequently,  appraisal  results  are  included  in  a  broader  look  at  elements  such  as  Contractor 
Perfonnance  Assessment  Reports.  This  broader  look  may  require  careful  timing  of  appraisal  visits 
so  that  the  appraisal  information  can  be  integrated  with  other  risk  determinants. 

3.13  CONSIDER  CONDUCTING  POST-AWARD  PROCESS  APPRAISALS 
What  to  Do 

In  the  RFP  package,  the  acquirer  should  reserve  the  right  to  execute  post-award  appraisals  using 
CMMI-DEV  focused  on  the  process  areas  that  are  determined  to  be  critical  to  the  program  at  any 
given  time.  The  acquirer  should  not  choose  CMMI  maturity  levels  as  the  scope  of  post-award 
appraisals;  instead,  it  should  follow  the  direction  given  in  this  guidebook  and  scope  any  appraisal 
to  those  process  areas  deemed  critical  and  proven  to  be  beneficial  to  the  success  of  the  program. 

At  the  appropriate  time,  the  acquirer  should  execute  a  SCAMPI  Class  B  appraisal  using  an 
authorized  SCAMPI  Class  B  appraisal  lead  and  a  team  trained  and  qualified  according  to  the 
SCAMPI  Class  B  appraisal  method.  Depending  on  the  model  and  organizational  scope,  this  type 
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of  appraisal  can  be  expected  to  be  completed  in  approximately  five  to  seven  working  days,  with 
only  a  portion  of  this  time  at  the  offeror’s  site. 

The  acquirer  should  make  sure  that  this  requirement  includes  subcontractors  so  that  the  entire 
supplier  team  may  be  appraised. 

When  to  Do  It 

Ideally,  it  is  best  to  reserve  the  option  for  post-award  appraisals  prior  to  RFP  release.  The  acquirer 
should  include  infonnation  on  the  time  frame  of  the  planned  post-award  appraisals  and  provide  a 
range  during  which  the  appraisals  could  be  planned  and  require  that  the  offeror  include  its 
proposed  dates  in  its  IMS. 

When  planning  the  second  post-award  appraisal,  it  may  be  appropriate  for  the  acquirer  to  schedule 
it  preceding  significant  milestones.  The  importance  of  different  processes  varies  throughout  the 
development  lifecycle.  Just  as  requirements  development,  requirements  management,  and  project 
planning  may  dominate  process  attention  in  the  early  phases,  increased  attention  to  configuration 
management,  verification,  and  validation  may  merit  greater  attention  later  in  the  lifecycle. 

How  to  Use  the  Results 

See  Section  4.4  on  page  30  and  its  associated  Appendix. 

3.14  CONSIDER  PROCESS-BASED,  OUTCOME-ORIENTED  AWARD  FEE 
DETERMINANTS 

To  facilitate  discussion  and  share  proven  incentive  strategies  across  the  entire  U.S.  Defense 
acquisition  workforce,  the  DoD  established  the  award  and  incentive  fees  community  of  practice 
(COP)  under  the  leadership  of  the  Defense  Acquisition  University  (DAU)  [DoD  2006d].  The  COP 
serves  as  the  repository  for  all  acquisition-related  materials,  including  policy  information,  training 
courses,  examples  of  good  award  fee  arrangements,  and  other  supporting  resources. 

What  to  Do 

An  approved  acquisition  strategy  may  result  in  award  fees  being  included  in  the  approved 
contract.  Process  improvement  planning  and  execution  can  be  part  of  award  fee  determination. 

The  acquirer  should  evaluate  effective  process  implementation  across  the  supplier  team  to 
mitigate  integration  risks.  An  evaluation  plan,  linked  to  award  fee  timing,  must  be  developed  and 
coordinated  with  stakeholders.  Agencies  like  DCMA  can  assist  with  such  efforts. 

When  to  Do  It 

The  first  decision  point  is  associated  with  assuring  that  the  acquisition  strategy  allows  award  fees. 
If  it  does,  the  best  time  to  introduce  process  discipline  as  an  award  fee  element  is  in  the  RFP. 

How  to  Use  the  Results 

Award  fees  give  the  acquirer  a  powerful  way  to  recognize  achievement,  or  failure,  in  risk 
mitigation.  The  results  provide  an  excellent  method  to  encourage  team  accomplishment  through  a 
reward  system.  It  provides  regular  checks  for  continued  progress  and  management  attention.  (See 
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Appendix  B,  “Systems  Engineering  in  an  Award  Fee  Plan,”  in  the  Guide  for  Integrating  Systems 
Engineering  into  DoD  Acquisition  Contracts,  for  further  guidance.) 
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Appendix  D:  Monitoring  Supplier  Process  Performance  After 
Contract  Award 


4.1  PROCESS  MONITORING  THROUGH  ARTIFACT  REVIEWS 
What  to  Do 

Joint  reviews  of  progress  are  frequent  and  familiar  to  most  acquirers.  Occasional  reviews  of  the 
process  quality  assurance  data  may  provide  excellent  insight  into  areas  that  may  pose  risks  to 
development.  Some  organizations  establish  separate  reviews  for  technical  performance,  cost  and 
schedule  (i.e.,  EVMS)  performance,  and  development  process  performance. 

When  to  Do  It 

It  is  desirable  to  link  process  reviews  with  other  related  activities,  such  as  ISO  audits  or  the 
company’s  independent  process  improvement  appraisals. 

How  to  Use  the  Results 

The  acquirer  should  use  the  results  to  manage  the  risks  to  ensure  successful  completion  of  the 
development  program. 

4.2  PROCESS  MONITORING  THROUGH  REVIEW  OF  PROCESS  METRICS 
What  to  Do 

Process  perfonnance  is  a  measure  of  the  actual  results  achieved  by  following  a  process.  These 
measures  can  be  used  to  determine  if  processes  are  performing  within  the  expected  bounds  set  by 
the  organization.  The  acquirer  should  request  documentation  from  the  contracted  organization  that 
includes  its  established  performance  baselines  and  performance  models  for  the  processes 
determined  to  be  critical  to  the  program.  Any  high  maturity  organization  should  have  this 
documentation  available.  Also,  throughout  the  life  of  the  program,  the  acquirer  should  request 
performance  data  periodically,  as  documented  by  the  supplier  for  effective  monitoring  of  these 
processes. 

When  to  Do  It 

Monitoring  process  performance  metrics  can  be  addressed  in  the  RFP.  Examples  of  process 
performance  monitoring  can  be  examined  in  pre-award  SCAMPI  appraisals  if  properly  scoped. 
Shortly  after  contract  award,  the  following  can  be  agreed  to: 

1 .  the  exact  process  performance  metrics 

2.  the  expected  amount  of  time  before  specific  metrics  would  be  available 

3.  how  often  reporting  of  those  metrics  will  occur. 

Monitoring  these  metrics  can  occur  throughout  the  life  of  the  program,  prioritized  based  on  risk,  if 
necessary,  to  properly  manage  the  acquirer’s  resources. 


SOFTWARE  ENGINEERING  INSTITUTE  |  61 


How  to  Use  the  Results 

Based  on  the  supplied  performance  baselines,  the  need  for  supplier  action  should  be  evident  when 
metrics  fall  outside  the  expected  bounds  of  performance.  The  need  for  supplier  action  is  most 
evident  for  suppliers  using  statistical  process  control.  Both  the  acquirer  and  supplier  can  jointly 
review  thresholds  and  confirm  that  they  are  understood  and  appropriate.  The  acquirer  should 
monitor  and  track  these  actions  to  ensure  they  come  to  appropriate  closure. 

4.3  PROCESS  MONITORING  THROUGH  PROCESS  REVIEWS 

The  acquirer  should  obtain  process  performance  reports  from  the  supplier  to  gain  insight  into  the 
gaps  that  the  supplier  has  identified  in  its  own  processes.  If  these  are  not  available,  the  acquirer 
should  consider  conducting  process  reviews,  potentially  using  DCMA. 

What  to  Do 

For  high  risk  efforts,  or  if  there  are  doubts  about  the  supplier  team’s  development  processes,  a 
high-level  evaluation  of  the  supplier  team’s  defined  processes  can  be  performed  within  a  few 
weeks  of  contract  award.  The  evaluation  provides  direct  information  concerning  the  intent  of  the 
supplier  team  to  fully  define  its  development  processes  and  integrate  them  into  a  cohesive 
development  process — activities  that  will  likely  cause  program  failure  if  not  attended  to.  A 
SCAMPI  Class  C  appraisal  is  an  effective,  quick,  low-cost  method  for  this  type  of  evaluation.  The 
initial  evaluation  can  be  followed  by  a  second,  more  in-depth  evaluation  of  the  integrated 
development  process  capability.  The  timing  of  this  second  evaluation  is  dependent  on  the 
situation,  and  the  SCAMPI  Class  B  appraisal  is  of  significant  value  in  these  cases. 

In  low  (process)  risk  efforts,  the  acquirer  may  elect  to  perform  just  the  in-depth  evaluation  at  an 
appropriate  time — perhaps  two  months  or  so  after  contract  award.  In  both  the  high  and  low 
(process)  risk  cases,  follow-on  evaluations  may  be  warranted  to  ensure  process  weaknesses  are 
addressed  appropriately  and  to  ensure  the  supplier  team’s  development  processes  do  not  degrade 
over  time 

When  to  Do  It 

One  review  is  conducted  shortly  after  contract  award,  when  the  initial  processes  have  been 
tailored  and  are  in  execution,  to  obtain  a  benchmark  of  process  performance.  A  second  review  can 
be  conducted  later  in  the  program,  possibly  around  the  same  time  as  a  major  program  decision 
point  (e.g.,  critical  design  review). 

In  addition  to  reviews,  the  acquirer  may  request  process  metrics  and  the  results  of  supplier  quality 
audits  of  its  development  processes  or  products. 

How  to  Use  the  Results 

The  acquirer  should  use  the  results  of  process  reviews,  process  audits,  or  process  metrics  to  ensure 
that  the  supplier’s  development  process  is  mature,  capable,  and  responsive  to  acquirer  needs.  Any 
process  interfaces  to  acquirer  processes  should  be  analyzed  and  understood.  Process  review  data 
should  provide  a  picture  of  firm  commitment  by  the  supplier  to  process  capability  and  continuous 
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improvement.  Process  gaps  and  findings  can  be  considered  for  their  risk  implications.  The  action 
plans  that  are  part  of  risk  mitigation  can  then  be  tracked  to  resolution. 

4.4  PROCESS  MONITORING  THROUGH  POST -AWARD  APPRAISALS 
What  to  Do 

The  acquirer  should  provide  instructions  on  approximate  time  frames  for  the  conduct  of  such 
appraisals.  In  general,  two  process  appraisals  are  beneficial.  The  acquirer  can  then  schedule  and 
conduct  an  appraisal  of  the  supplier  team. 

To  exercise  this  method,  it  is  not  necessary  to  have  appraisal  expertise.  Most  often,  the  acquirer 
can  work  with  an  appraisal  lead  trained  and  qualified  to  conduct  the  appraisal  desired.  The 
appraisal  lead  works  with  the  sponsor  to  clarify  the  needs  and  requirements  for  the  appraisal.  The 
appraisal  lead  may  then  recommend  alternative  approaches  to  fulfill  the  requirements.  Once  an 
approach  is  decided,  the  appraisal  lead  begins  to  develop  detailed  plans  and  estimates  for  approval 
by  the  sponsor  (i.e.,  the  acquirer). 

The  acquirer  should  be  careful  when  choosing  an  appraisal  lead.  Many  suppliers  employ  internal 
staff  who  are  trained  in  CMMI  and  are  qualified  to  lead  appraisals.  Many  appraisal  leads  also 
offer  process  improvement  consulting  services  to  help  client  organizations  achieve  desired 
process  maturity.  Use  of  appraisal  leads  employed  by  suppliers,  either  directly  or  as  consultants, 
should  be  avoided.  While  such  appraisal  leads  offer  the  advantage  of  familiarity  with  the 
organizations  being  appraised,  this  advantage  is  offset  by  the  potential  conflict  of  interest.  It  is 
best  for  the  acquirer  to  choose  an  appraisal  lead  who  is  independent  from  the  supplier 
organization  and  has  experience  within  the  technology  domain  of  the  project. 

The  acquirer  should  clearly  communicate  the  goals  to  the  appraisal  lead  and  ensure  that  the 
appraisal  lead  understands  the  following: 

•  the  process  areas  that  are  critical  to  the  project 

•  the  scope  of  the  appraisal  (Unlike  most  appraisals  that  examine  the  processes  within  an 
organization,  this  appraisal  is  examining  processes  only  within  the  program .) 

•  the  current  status  of  the  program 

•  the  motivation  for  the  appraisal  (e.g.,  to  encourage  rapid  process  application  on  a  newer 
project,  to  check  for  compliance  commitments  made  during  bidding,  or  to  assess  problem 
areas  within  the  program) 

•  the  expectations  for  reporting  (e.g.,  a  report  of  process  capability  in  selected  process  areas,  a 
report  of  process  strengths  and  weaknesses,  or  a  report  of  risks  arising  from  processes) 

•  the  cost  and  schedule  targets  for  the  appraisal 

Based  on  this  information,  the  appraisal  lead  recommends  the  type  of  appraisal  (i.e.  SCAMPI 
Class  B,  or  Class  C),  and  develops  an  appraisal  plan  for  review  and  approval. 

The  appraisal  lead  forms  an  appraisal  team  consisting  of  suitably  qualified  members.  Including 
both  qualified  staff  members  from  the  acquirer’s  organization  and  qualified  supplier  staff 
members  on  the  appraisal  team  is  helpful.  The  supplier  team  members  understand  the  supplier’s 
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processes  and  can  aid  the  team  in  finding  and  interpreting  data.  The  acquirer  team  members 
understand  the  needs  of  the  acquirer,  and  can  focus  the  team  on  the  aspects  most  critical  to  the 
program.  Once  the  team  is  formed,  team  training  is  conducted  to  establish  the  appraisal  ground 
rules  and  make  appropriate  assignments  to  team  members. 

When  to  Do  It 

When  to  execute  the  post-award  appraisal  can  be  driven  by  several  factors,  including  when  the 
resulting  work  products  related  to  the  processes  in  question  will  be  present  and  where  in  the 
development  lifecycle  the  processes  are  expected  to  be  implemented.  Depending  on  the  program, 
this  appraisal  might  also  be  an  opportunity  to  look  forward  in  the  development  lifecycle  and 
examine  processes  not  previously  examined.  Typically,  post-award  SCAMPI  appraisals  are  not 
executed  prior  to  six  months  after  contract  award  unless  there  is  high  risk  in  processes  related  to 
early  stages  of  the  program’s  development  lifecycle. 

The  timing  of  supplier  team  appraisals  depends  on  the  motivation  for  the  appraisal.  If  the 
motivation  for  the  appraisal  is  to  encourage  rapid  process  application  on  a  newer  project,  the 
appraisal  should  be  conducted  soon  after  contract  award.  The  acquirer  should  allow  enough  time 
to  elapse  to  enable  the  supplier  to  start  the  program,  to  coordinate  with  subcontractors,  and  to 
produce  some  process  artifacts  for  evaluation.  Six  months  after  contract  award  is  usually  a  good 
time  for  this  type  of  appraisal;  however,  not  all  processes  may  be  implemented  at  this  early  stage 
of  the  program. 

Processes  such  as  project  planning,  risk  management,  project  monitoring  and  control, 
requirements  development,  and  requirements  management  should  be  evident.  If  the  motivation  for 
the  appraisal  is  to  check  for  compliance  commitments  made  during  bidding,  this  appraisal  can  be 
performed  at  any  time  during  the  project.  The  process  areas  appraised  should  be  consistent  with 
the  current  lifecycle  phase.  If  the  motivation  for  the  appraisal  is  to  assess  problem  areas  within  the 
program,  this  appraisal  should  be  done  as  soon  as  process  issues  become  evident.  Early 
intervention  may  contain  the  impact  of  the  problems  and  can  also  indicate  to  the  supplier  that  the 
acquirer  is  serious  about  processes.  The  supplier  may  therefore  be  more  proactive  in  preventing 
future  process  problems. 

How  to  Use  the  Results 

For  process  weaknesses,  risks,  or  process  implementation  issues  identified  by  the  appraisal,  the 
acquirer  should  ask  the  following  questions: 

1 .  What  is  the  impact  of  this  on  the  program? 

2.  What  suitable  methods  are  available  to  address  it? 

3.  When  must  it  be  addressed? 

4.  Whose  responsibility  is  it  to  address  it? 

a.  Is  the  supplier  willing  to  address  it? 

b.  Is  the  supplier  contractually  obligated  to  address  it? 

c.  Is  a  contract  change  necessary  to  address  it? 

d.  Should  incentives  be  used  to  encourage  the  supplier  to  address  it? 
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5.  How  will  the  status  of  the  issue  and  the  plan  to  address  it  be  monitored  and  reported? 

6.  Is  this  issue  symptomatic  of  other  problems? 
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Appendix  E:  Mapping  Between  DAG  Chapter  4  Processes 
and  CMMI  Process  Areas 


Table  7:  Mapping  between  DAG  Chapter  4  Processes  and  CMMI  Process  Areas 


Ch.  4,  Defense  Acquisition  Guidebook 

CMMI-DEV,  VI  .2,  2006 

Technical  Processes 

Requirements  Development 

Engineering:  Requirements  Development 

Logical  Analysis 

Engineering:  Requirements  Development 

Design  Solution 

Engineering:  Technical  Solution 

Implementation 

Engineering:  Technical  Solution 

Integration 

Engineering:  Product  Integration 

Verification 

Engineering:  Verification 

Validation 

Engineering:  Validation 

Transition 

Engineering:  Product  Integration 

Technical  Management  Processes 

Decision  Analysis 

Support:  Decision  Analysis  and  Resolution 

Technical  Planning 

Project  Management:  Project  Planning 

Technical  Assessment 

Support:  Measurement  and  Analysis 

Project  Management:  Project  Monitoring  and  Control 

Requirements  Management 

Engineering:  Requirements  Management 

Risk  Management 

Project  Management:  Risk  Management 

Configuration  Management 

Support:  Configuration  Management 

Technical  Data  Management 

Project  Management:  Project  Planning 

Interface  Management 

Engineering:  Product  Integration 
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Appendix  F:  Acronyms 


ADS 

Appraisal  Disclosure  Statement 

ANSI 

American  National  Standards  Institute 

BAFO 

best  and  final  offer 

BoE 

basis  of  estimate 

CAR 

Causal  Analysis  and  Resolution 

CDRL 

contract  data  requirements  list 

CM 

Configuration  Management 

CMMI 

Capability  Maturity  Model  Integration 

CMMI-DEV 

CMMI  for  Development 

COP 

community  of  practice 

DAG 

Defense  Acquisition  Guidebook 

DAL 

data  accession  list 

DAR 

Decision  Analysis  and  Resolution 

DAU 

Defense  Acquisition  University 

DCMA 

Defense  Contract  Management  Agency 

DoD 

Department  of  Defense 

EN 

evaluation  notice 

EVMS 

earned  value  management  system 

FAR 

Federal  Acquisition  Regulation 

FFRDC 

federally  funded  research  and  development  center 

FPD 

final  proposal  document 

IBR 

initial  baseline  review 

IDE 

integrated  development  environment 

IEC 

International  Electrotechnical  Commission 

IEEE 

Institute  of  Electrical  and  Electronic  Engineers 

IMP 

integrated  master  plan 
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IMS 

integrated  master  schedule 

IPM 

Integrated  Project  Management 

IPPD 

integrated  product  and  process  development 

IPT 

integrated  project  team 

ISO 

International  Organisation  for  Standardisation 

JPO 

Joint  Program  Office 

MA 

Measurement  and  Analysis 

NA 

not  applicable 

NASA 

National  Aeronautics  and  Space  Administration 

OID 

Organizational  Innovation  and  Deployment 

OPD 

Organizational  Process  Definition 

OPF 

Organizational  Process  Focus 

OPP 

Organizational  Process  Performance 

OT 

Organizational  Training 

PA 

process  area 

PARS 

Published  Appraisal  Report  Site 

PI 

Product  Integration 

PII 

process  implementation  indicators 

PMC 

Project  Monitoring  and  Control 

PMO 

Program  Management  Office 

PP 

Project  Planning 

PPQA 

Process  and  Product  Quality  Assurance 

QPM 

Quantitative  Project  Management 

RD 

Requirements  Development 

REQM 

Requirements  Management 

RFI 

request  for  information 

RFP 

request  for  proposal 

RMP 

risk  management  plan 

RSKM 

Risk  Management 
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SAM 

Supplier  Agreement  Management 

SCAMPI 

Standard  CMMI  Appraisal  Method  for  Process  Improvement 

SDP 

software  development  plan 

SE 

systems  engineering 

SE/SW 

systems  engineering  and  software  engineering 

SEC 

software  engineering  center 

SEI 

Software  Engineering  Institute 

SEP 

systems  engineering  plan 

ss 

supplier  sourcing 

SSA 

software  support  activities 

sw 

software  engineering 

SW-CMM 

Capability  Maturity  Model  for  Software 

TS 

Technical  Solution 

TSP 

Team  Software  Process 

VAL 

Validation 

VER 

Verification 

WBS 

work  breakdown  structure 
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